Multipath and Anaconda
======================

:Authors:
   Ales Kozumplik <akozumpl@redhat.com>

Introduction
------------

If there are two block devices in your /dev for which udev reports the same
'ID_SERIAL' then you can create a certain device mapper device which arbitrarily
uses those devices to access the physical device. And that is Multipath [1].

For instance, suppose there are::

  /dev/sda, with ID_SERIAL of 20090ef12700001d2, and
  /dev/sdb, with the same ID_SERIAL.

Those are probably some adapters in the system that just connect your box to a
storage area network (SAN) somewhere. There are perhaps two cables, one for sda,
one for sdb, and if one of the cables gets cut the other can still transmit
data. Normally the system won't recognize that sda and sdb have this special
relation to each other, but by creating a suitable device map using multipath
tools [2] we can create a DM device /dev/mapper/mpatha and use it for storing
and retrieving data.

The device mapper then automatically routes IO requests to /dev/mapper/mpatha to
either sda or sdb depending on the load of the line or network congestion on the
particular network etc.

The nomenclature I will use here is:
- 'multipath device' for the smart /dev/mapper/mpathX device.
- 'multipath member device' for the '/dev/sdX' devices. Also 'a path'.


What is expected from Anaconda
------------------------------

Anaconda is expected to:
- detect that there are multipath devices present
- coalesce all relevant (e.g. exclusiveDisks) multipath devices.
- only let the user interact with the multipath devices in filtering,
  cleardiskssel and partition screen, that is once we know 'sdc' and 'sdd' are
  part of 'mpathb' show only 'mpathb' and never the paths.
- install bootloader and boot from an mpath device
- make it happen so all the multipath devices (carrying or not the root
  filesystem) we used for installation are correctly coalesced in the booted
  system. This is achieved by generating a suitable /etc/multipath.conf and
  writing it into sysroot.
- be able to refer to mpath devices from kickstart, either by name like 'mpatha'
  or by their id like 'disk/by-id/scsi-20090ef12700001d2'


How Anaconda handles multipath
------------------------------

To detect presence of multipath devices we rely on multipath tools. The same we
do for coalescing, see pyanaconda/storage/devicelibs/mpath.py, the file that
provides some abstraction from mpath tools. During the device scan we use the
'multipath -d' output to find out what devices are going to end up as multipath
members. The MultipathTopology object also enhances the multipath member's udev
dictionaries with 'ID_FS_TYPE' set to 'multipath_member' (yes, this is a hack
surviving from the original mpath implementation, and righteous is he who
eradicates it). This information is picked up by DeviceTree when populating
itself. Meaning, if 'sda' and 'sdb' are multipath member devices DeviceTree
gives them MultipathMember format and creates one MultipathDevice for them (we
know its name from 'multipath -d'). We end up with:

DiskDevice 'sda', format 'MultipathMember'
DiskDevice 'sdb', format 'MultipathMember'
MultipathDevice 'mpatha', parents are 'sda' and 'sdb'.

From then on, Anaconda only deals with the MultipathDevice and generally leaves
anything with 'MultipathMember' format alone (understand, this is an inert
format that really is not there but we use it just to mark the device as
"useless beyond a multipath member", kind of like MDRaidMember).

Partition happens over the multipath device and during the preinstallconfig step
/mnt/sysimage/etc/multipath.conf is created and filled with information about
the coalesced devices. This is handled in the Storage.write() method. It is
important this file and /etc/multipath/wwids (autogenerated by mpath tools)
make it to the sysimage before the dracut image is generated.


Debugging multipath bugs
-------------------------

Unlike with iSCSI, to reproduce a multipath bug one does not need the same
specific hardware as the reporter. Just found any box connected to a multipathed
SAN and you are fine (at the moment, connecting to the same iSCSI target through
its IPv4 and IPv6 address also produces a multipathed device).

On top of that, much of the necessary information is already included in the
anaconda logs or can be easily extracted from the reporter. The things to
particularly look at are:

- storage.log, the output around 'devices to scan for multipath' and 'devices
  post multipath scan'. The latter shows a triple with regular disks, disks
  comprising multipath devices and partitions. This helps you quickly find out
  what the target system is about.

- this information is also in program.log's calls to 'multipath' [3]. If mpath
  devices are mysteriously appearing/disappearing between filtering and
  partitioning screens look at those. 'multipath -ll' is called to display
  currently coalesced mpath devices, 'multipath -d' is called to show the mpath
  devices that would be coalesced if we ran 'multipath' now. This is exploited
  by the device filtering screen.


Future of multipath in Anaconda
--------------------------------

Overall as of RHEL6.2, the shape of multipath in Anaconda is good and what's
more important it is flexible enough to sustain new RFEs and bugs. Those are
however bugs that I expect to appear sometime soon:

- enable or disable mpath_friendly_names in kickstart. Disabling friendly names
  just means the mpath devices are called by their wwid,
  e.g. /dev/mapper/360334332345343234, not '/dev/mapper/mpathc'. This is
  straightforward to implement.
- extend support for mpath devices in kickstart in general. Currently mpath
  devices should be accepted in most commands but I am sure there will be corner
  cases. Difficulty medium.
- [rawhide] stop extending the udev info dictionary with 'ID_FS_TYPE' and
  'ID_MPATH_NAME'. Doing it this way is asking for the trouble if a dictionary
  of particular mpath device is reloaded from udev without running it through
  the MultipathTopology object as it will miss those entries (and DeviceTree
  depends on them a lot). Difficulty hard, but includes a lot of pleasant
  refactoring.
- Improve support for multipathing iSCSI devices. Someone might ask for it one
  day (in fact, with the NIC bounding they already did), and it will make mpath
  debugging possible on any virt machine with multiple virt NICs.


.. [1] http://akozumpl.fedorapeople.org/archive/Multipass.jpg
.. [2] http://christophe.varoqui.free.fr/
.. [3] 'man 8 multipath'

