1 Introduction 1 Introduction
2 ============ 2 ============
3 3
4 ATA over Ethernet is a network protocol that p 4 ATA over Ethernet is a network protocol that provides simple access to
5 block storage on the LAN. 5 block storage on the LAN.
6 6
7 http://support.coraid.com/documents/AoEr11.t 7 http://support.coraid.com/documents/AoEr11.txt
8 8
9 The EtherDrive (R) HOWTO for 2.6 and 3.x kerne 9 The EtherDrive (R) HOWTO for 2.6 and 3.x kernels is found at ...
10 10
11 http://support.coraid.com/support/linux/Ethe 11 http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO.html
12 12
13 It has many tips and hints! Please see, espec 13 It has many tips and hints! Please see, especially, recommended
14 tunings for virtual memory: 14 tunings for virtual memory:
15 15
16 http://support.coraid.com/support/linux/Ethe 16 http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO-5.html#ss5.19
17 17
18 The aoetools are userland programs that are de 18 The aoetools are userland programs that are designed to work with this
19 driver. The aoetools are on sourceforge. 19 driver. The aoetools are on sourceforge.
20 20
21 http://aoetools.sourceforge.net/ 21 http://aoetools.sourceforge.net/
22 22
23 The scripts in this Documentation/admin-guide/ 23 The scripts in this Documentation/admin-guide/aoe directory are intended to
24 document the use of the driver and are not nec 24 document the use of the driver and are not necessary if you install
25 the aoetools. 25 the aoetools.
26 26
27 27
28 Creating Device Nodes 28 Creating Device Nodes
29 ===================== 29 =====================
30 30
31 Users of udev should find the block device n 31 Users of udev should find the block device nodes created
32 automatically, but to create all the necessa 32 automatically, but to create all the necessary device nodes, use the
33 udev configuration rules provided in udev.tx 33 udev configuration rules provided in udev.txt (in this directory).
34 34
35 There is a udev-install.sh script that shows 35 There is a udev-install.sh script that shows how to install these
36 rules on your system. 36 rules on your system.
37 37
38 There is also an autoload script that shows 38 There is also an autoload script that shows how to edit
39 /etc/modprobe.d/aoe.conf to ensure that the 39 /etc/modprobe.d/aoe.conf to ensure that the aoe module is loaded when
40 necessary. Preloading the aoe module is pre 40 necessary. Preloading the aoe module is preferable to autoloading,
41 however, because AoE discovery takes a few s 41 however, because AoE discovery takes a few seconds. It can be
42 confusing when an AoE device is not present 42 confusing when an AoE device is not present the first time the a
43 command is run but appears a second later. 43 command is run but appears a second later.
44 44
45 Using Device Nodes 45 Using Device Nodes
46 ================== 46 ==================
47 47
48 "cat /dev/etherd/err" blocks, waiting for er 48 "cat /dev/etherd/err" blocks, waiting for error diagnostic output,
49 like any retransmitted packets. 49 like any retransmitted packets.
50 50
51 "echo eth2 eth4 > /dev/etherd/interfaces" te 51 "echo eth2 eth4 > /dev/etherd/interfaces" tells the aoe driver to
52 limit ATA over Ethernet traffic to eth2 and 52 limit ATA over Ethernet traffic to eth2 and eth4. AoE traffic from
53 untrusted networks should be ignored as a ma 53 untrusted networks should be ignored as a matter of security. See
54 also the aoe_iflist driver option described 54 also the aoe_iflist driver option described below.
55 55
56 "echo > /dev/etherd/discover" tells the driv 56 "echo > /dev/etherd/discover" tells the driver to find out what AoE
57 devices are available. 57 devices are available.
58 58
59 In the future these character devices may di 59 In the future these character devices may disappear and be replaced
60 by sysfs counterparts. Using the commands i 60 by sysfs counterparts. Using the commands in aoetools insulates
61 users from these implementation details. 61 users from these implementation details.
62 62
63 The block devices are named like this:: 63 The block devices are named like this::
64 64
65 e{shelf}.{slot} 65 e{shelf}.{slot}
66 e{shelf}.{slot}p{part} 66 e{shelf}.{slot}p{part}
67 67
68 ... so that "e0.2" is the third blade from t 68 ... so that "e0.2" is the third blade from the left (slot 2) in the
69 first shelf (shelf address zero). That's th 69 first shelf (shelf address zero). That's the whole disk. The first
70 partition on that disk would be "e0.2p1". 70 partition on that disk would be "e0.2p1".
71 71
72 Using sysfs 72 Using sysfs
73 =========== 73 ===========
74 74
75 Each aoe block device in /sys/block has the 75 Each aoe block device in /sys/block has the extra attributes of
76 state, mac, and netif. The state attribute 76 state, mac, and netif. The state attribute is "up" when the device
77 is ready for I/O and "down" if detected but 77 is ready for I/O and "down" if detected but unusable. The
78 "down,closewait" state shows that the device 78 "down,closewait" state shows that the device is still open and
79 cannot come up again until it has been close 79 cannot come up again until it has been closed.
80 80
81 The mac attribute is the ethernet address of 81 The mac attribute is the ethernet address of the remote AoE device.
82 The netif attribute is the network interface 82 The netif attribute is the network interface on the localhost
83 through which we are communicating with the 83 through which we are communicating with the remote AoE device.
84 84
85 There is a script in this directory that for 85 There is a script in this directory that formats this information in
86 a convenient way. Users with aoetools shoul 86 a convenient way. Users with aoetools should use the aoe-stat
87 command:: 87 command::
88 88
89 root@makki root# sh Documentation/admin-gu 89 root@makki root# sh Documentation/admin-guide/aoe/status.sh
90 e10.0 eth3 up 90 e10.0 eth3 up
91 e10.1 eth3 up 91 e10.1 eth3 up
92 e10.2 eth3 up 92 e10.2 eth3 up
93 e10.3 eth3 up 93 e10.3 eth3 up
94 e10.4 eth3 up 94 e10.4 eth3 up
95 e10.5 eth3 up 95 e10.5 eth3 up
96 e10.6 eth3 up 96 e10.6 eth3 up
97 e10.7 eth3 up 97 e10.7 eth3 up
98 e10.8 eth3 up 98 e10.8 eth3 up
99 e10.9 eth3 up 99 e10.9 eth3 up
100 e4.0 eth1 up 100 e4.0 eth1 up
101 e4.1 eth1 up 101 e4.1 eth1 up
102 e4.2 eth1 up 102 e4.2 eth1 up
103 e4.3 eth1 up 103 e4.3 eth1 up
104 e4.4 eth1 up 104 e4.4 eth1 up
105 e4.5 eth1 up 105 e4.5 eth1 up
106 e4.6 eth1 up 106 e4.6 eth1 up
107 e4.7 eth1 up 107 e4.7 eth1 up
108 e4.8 eth1 up 108 e4.8 eth1 up
109 e4.9 eth1 up 109 e4.9 eth1 up
110 110
111 Use /sys/module/aoe/parameters/aoe_iflist (o 111 Use /sys/module/aoe/parameters/aoe_iflist (or better, the driver
112 option discussed below) instead of /dev/ethe 112 option discussed below) instead of /dev/etherd/interfaces to limit
113 AoE traffic to the network interfaces in the 113 AoE traffic to the network interfaces in the given
114 whitespace-separated list. Unlike the old c 114 whitespace-separated list. Unlike the old character device, the
115 sysfs entry can be read from as well as writ 115 sysfs entry can be read from as well as written to.
116 116
117 It's helpful to trigger discovery after sett 117 It's helpful to trigger discovery after setting the list of allowed
118 interfaces. The aoetools package provides a 118 interfaces. The aoetools package provides an aoe-discover script
119 for this purpose. You can also directly use 119 for this purpose. You can also directly use the
120 /dev/etherd/discover special file described 120 /dev/etherd/discover special file described above.
121 121
122 Driver Options 122 Driver Options
123 ============== 123 ==============
124 124
125 There is a boot option for the built-in aoe 125 There is a boot option for the built-in aoe driver and a
126 corresponding module parameter, aoe_iflist. 126 corresponding module parameter, aoe_iflist. Without this option,
127 all network interfaces may be used for ATA o 127 all network interfaces may be used for ATA over Ethernet. Here is a
128 usage example for the module parameter:: 128 usage example for the module parameter::
129 129
130 modprobe aoe_iflist="eth1 eth3" 130 modprobe aoe_iflist="eth1 eth3"
131 131
132 The aoe_deadsecs module parameter determines 132 The aoe_deadsecs module parameter determines the maximum number of
133 seconds that the driver will wait for an AoE 133 seconds that the driver will wait for an AoE device to provide a
134 response to an AoE command. After aoe_deads 134 response to an AoE command. After aoe_deadsecs seconds have
135 elapsed, the AoE device will be marked as "d 135 elapsed, the AoE device will be marked as "down". A value of zero
136 is supported for testing purposes and makes 136 is supported for testing purposes and makes the aoe driver keep
137 trying AoE commands forever. 137 trying AoE commands forever.
138 138
139 The aoe_maxout module parameter has a defaul 139 The aoe_maxout module parameter has a default of 128. This is the
140 maximum number of unresponded packets that w 140 maximum number of unresponded packets that will be sent to an AoE
141 target at one time. 141 target at one time.
142 142
143 The aoe_dyndevs module parameter defaults to 143 The aoe_dyndevs module parameter defaults to 1, meaning that the
144 driver will assign a block device minor numb 144 driver will assign a block device minor number to a discovered AoE
145 target based on the order of its discovery. 145 target based on the order of its discovery. With dynamic minor
146 device numbers in use, a greater range of Ao 146 device numbers in use, a greater range of AoE shelf and slot
147 addresses can be supported. Users with udev 147 addresses can be supported. Users with udev will never have to
148 think about minor numbers. Using aoe_dyndev 148 think about minor numbers. Using aoe_dyndevs=0 allows device nodes
149 to be pre-created using a static minor-numbe 149 to be pre-created using a static minor-number scheme with the
150 aoe-mkshelf script in the aoetools. 150 aoe-mkshelf script in the aoetools.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.