BlueDBM Setup Instructions

Back to BlueDBM
May 1, 2015M

1. Hardware Setup

• Required equipment list:

  1. Host PC w/ x8 PCIe slot or larger
  2. Xilinx VC707 FPGA development board
  3. BlueDBM Flash storage card (up to two per node) + Power adapter
  4. SATA crossover cables (2 per link)
  5. PCIe x8 riser cable

• Setup steps
  1. The two flash cards plug into each of the FMC ports on the VC707
  2. The VC707 plugs into the host server's PCIe slot, via riser cable if required
  3. Power adapters for the VC707 and two flash cards should be plugged in. Note: The green LEDS will light up if the flash card has been correctly power sequenced. If not, the power sequencer needs to be programmed.
  4. The software in the repository assumes server names are named bdbm## (e.g., bdbm00, bdbm01...).
  
  
• Network setup
  
  
  1. When holding the storage node horizontally, with the PCIe to the right and flash card to the left, with the fan facing towards you, the network ports are numbered like the above image.
     
  2. The default bar topology router expects the following connection:

     node n's port 0 connects to node n+1's port 3

     node n's port 1 connects to node n+1's port 2

2. Getting the Code

• The repository for BlueDBM is currently:
  https://github.com/sangwoojun/bluedbm
  

• Once you clone the repository, run "init_xbsv.sh". This will clone the repositories of the tools required to build and run BlueDBM. (connectal and related tools)

• Each project developed using BlueDBM are stored in the directory "projects". Each project can either be built for the bluesim environment or the vc707 hardware by running:
  make build.bluesim or
  make build.vc707 respectively

• Note: When running a bluesim simulation for multiple nodes with a network, multiple copies of the bluesim directory needs to be made and run separately. Consult "run_bsim.sh" in projects/remoteaccess.

3. Node Environment Setup

• By default BlueDBM expects the host IDs to be named bdbm## (e.g., bdbm00, bdbm01...). If this is not the case, the enviornment variable BDBM_ID can be set to a numeric value (e.g., BDBM_ID=1).
  Caution: The host IDs must match the connections made with the network links. See section "Network Setup"

• The PCIe drivers for connectal can be found in tools/connectal/drivers.
  To install the drivers, go to tools/connectal/drivers/pcieportal and run:
  make
  sudo make install
  sudo make insmod
  This process needs to be done only once.

4. Programming and Running

• Programming the FPGA involves first clearing the flash controller chip, clearing the Vertex FPGA, programming the flash controller chip, and then programming the Vertex FPGA.
  The required scripts can be downloaded from the link below:
  bdbmscripts.tgz

• Modify program-all.tcl and program2.sh to match the location of your "vc707" directory, which is the result of running make build.vc707 in the project directory.
  And then run ./program2.sh

• Note: New PCIe devices are enumerated by the BIOS only during the boot sequence. Therefore, if you are programming an FPGA which did not have BlueDBM on it already, you need to reboot the host server.
  You can check for the reboot requirement by running lspci | grep 1be7.
  If you see Memory controller: Device 1be7:c100 as part of the output, you probably don't need to reboot.

• Run your ./vc707/bin/ubuntu.exe

Contact: wjun _at_ csail _dot_ mit.edu