README 4.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140
  1. INTRODUCTION
  2. ------------
  3. The vblade is the virtual EtherDrive (R) blade, a program that makes a
  4. seekable file available over an ethernet local area network (LAN) via
  5. the ATA over Ethernet (AoE) protocol.
  6. The seekable file is typically a block device like /dev/md0 but even
  7. regular files will work. When vblade exports the block storage over
  8. AoE it becomes a storage target. Another host on the same LAN can
  9. access the storage if it has a compatible aoe kernel driver.
  10. BUILDING
  11. --------
  12. The following command should build the vblade program on a Linux-based
  13. system:
  14. make
  15. For FreeBSD systems, include an extra parameter like so:
  16. make PLATFORM=freebsd
  17. EXAMPLES
  18. --------
  19. There is a "vbladed" script that daemonizes the program and sends its
  20. output to the logger program. Make sure you have logger installed if
  21. you would like to run vblade as a daemon with the vbladed script.
  22. ecashin@kokone vblade$ echo 'I have logger' | logger
  23. ecashin@kokone vblade$ tail -3 /var/log/messages
  24. Feb 8 14:52:49 kokone -- MARK --
  25. Feb 8 15:12:49 kokone -- MARK --
  26. Feb 8 15:19:56 kokone logger: I have logger
  27. Here is a short example showing how to export a block device with a
  28. vblade. (This is a loop device backed by a sparse file, but you could
  29. use any seekable file instead of /dev/loop7.)
  30. ecashin@kokone vblade$ make
  31. cc -Wall -c -o aoe.o aoe.c
  32. cc -Wall -c -o linux.o linux.c
  33. cc -Wall -c -o ata.o ata.c
  34. cc -o vblade aoe.o linux.o ata.o
  35. ecashin@kokone vblade$ su
  36. Password:
  37. root@kokone vblade# modprobe loop
  38. root@kokone vblade# dd if=/dev/zero bs=1k count=1 seek=`expr 1024 \* 4096` of=bd
  39. -file
  40. 1+0 records in
  41. 1+0 records out
  42. 1024 bytes transferred in 0.009901 seconds (103423 bytes/sec)
  43. root@kokone vblade# losetup /dev/loop7 bd-file
  44. root@kokone vblade# ./vblade 9 0 eth0 /dev/loop7
  45. ioctl returned 0
  46. 4294968320 bytes
  47. pid 16967: e9.0, 8388610 sectors
  48. Here's how you can use the Linux aoe driver to access the storage from
  49. another host on the LAN.
  50. ecashin@kokone ecashin$ ssh makki
  51. Last login: Mon Feb 7 10:25:04 2005
  52. ecashin@makki ~$ su
  53. Password:
  54. root@makki ecashin# modprobe aoe
  55. root@makki ecashin# aoe-stat
  56. e9.0 eth1 up
  57. root@makki ecashin# mkfs -t ext3 /dev/etherd/e9.0
  58. mke2fs 1.35 (28-Feb-2004)
  59. ...
  60. Creating journal (8192 blocks): done
  61. Writing superblocks and filesystem accounting information: done
  62. This filesystem will be automatically checked every 24 mounts or
  63. 180 days, whichever comes first. Use tune2fs -c or -i to override.
  64. root@makki ecashin# mkdir /mnt/e9.0
  65. root@makki ecashin# mount /dev/etherd/e9.0 /mnt/e9.0
  66. root@makki ecashin# echo hooray > /mnt/e9.0/test.txt
  67. root@makki ecashin# cat /mnt/e9.0/test.txt
  68. hooray
  69. Remember: be as careful with these devices as you would with /dev/hda!
  70. Jumbo Frame Compatibility
  71. -------------------------
  72. Vblade can use jumbo frames provided your initiator is jumbo frame
  73. capable. There is one small configuration gotcha to consider
  74. to avoid having the vblade kernel frequently drop frames.
  75. Vblade uses a raw socket to perform AoE. The linux kernel will
  76. only buffer a certain amount of data for a raw socket. For 2.6
  77. kernels, this value is managed through /proc:
  78. root@nai aoe# grep . /proc/sys/net/core/rmem_*
  79. /proc/sys/net/core/rmem_default:128000
  80. /proc/sys/net/core/rmem_max:128000
  81. rmem_max is the max amount a user process may expand the receive
  82. buffer to -- through setsockopt(...) -- and rmem_default is, as you
  83. might expect, the default.
  84. The gotcha is that this amount to buffer does not relate
  85. to the amount of user data buffered, but the amount of
  86. real data buffered. As an example, the Intel GbE controller
  87. must be given 16KB frames to use an MTU over 8KB.
  88. For each received frame, the kernel must be able to buffer
  89. 16KB, even if the aoe frame is only 60 bytes in length.
  90. The linux aoe initiator will use 16 outstanding frames when
  91. used with vblade. A good default for ensuring frames are
  92. not dropped is to allocate 16KB for 17 frames:
  93. for f in /proc/sys/net/core/rmem_*; do echo $((17 * 16 * 1024)) >$f; done
  94. Be sure to start vblade after changing the buffering defaults
  95. as the buffer value is set when the socket is opened.
  96. AoE Initiator Compatibility
  97. ---------------------------
  98. The Linux aoe driver for the 2.6 kernel is compatible if you use
  99. aoe-2.6-7 or newer. You can use older aoe drivers but you will only
  100. be able to see one vblade per MAC address.
  101. Contrib Patches
  102. ---------------
  103. see contrib/README
  104. Kvblade
  105. -------
  106. While vblade runs as a userland process (like "ls" or "vi"), there
  107. is another program that runs inside the kernel. It is called
  108. kvblade. It is alpha software.