How To Improve Lustre Performance with Benchmark Analysics

Lustre is designed for file system scalability, high-performance, and high-availability.

The Goal of the Benchmark

To observer luster file system health ​ and ​ its supporting infrastructure ​ , we are running
benchmarking on the luster file system. These methods are intended to ​ exercise the
infrastructure​ to ​ identify faults​ and ​ enhance the performance ​ rather than just monitor.
The goal is to ensure the system is running according to the specification and defects have
been identified and eliminated.

Lustre is a virtual file system running in software. So it depends upon capabilities of
underlying platform(server, storage, network, OS, Lustre software)

What is LUN.
The storage is subdivided into volumes or LUNs. Linux representing a Lustre storage target
MDT and OST.

JBOD – Just Bunch of Disk
These storage systems typically comprise ​ multi-ported enclosures, ​ each containing an ​ array
of disks​ or other ​ persistent storage devices​ .

Establish the base operating platform

1. Server, storage, network physical integration cabling, and power.
2. Firmware update
3. BIOS Setting
4. storage configuration
5. OS Installation and configuration
6. Device driver installation
7. Application Software Installation(Including lustre)

Benchmarking of Lustre component

1. network, server, storage.
2. Benchmark the network
3. Benchmark ​ individual storage devices or LUNs.
4. Benchmark the OSS servers.
5. Benchmark the MDS servers.
6. Userspace/Application Benchmark

once upgraded respective driver then checks with the benchmark.
check everything we have the latest driver. once updated check the performance.

when the MDS filename lookup is complete and user and client have permission to access
and/or create the file, either the ​ layout of the existing file​ is returned to the client or a new
the file is created on the behalf of the client if requested. For read or write operation, the client
then interpret the file layout in the Logical volume(LOV)layer, which maps the file logical offset
and size to one or more objects.
Stripe Vs Filesize

small files have less stripe.
Large files have a higher stripe.

Imporant Lustre configuration file

/etc/modprobe.d/lnet.conf -> Lustre Kernel Module configuration file.
/etc/modprobe.d/ko2iblnd.conf
/etc/modprobe.d/ksocklnd.conf - client router side.
In the kernel Module tune the parameter.
Monitor/Change Parameter.
Common Parameter.
lfs getstripe
/proc/fs/lustre/osc/*max_rpcs_in_fight
/proc/fs/lustre/obdfileter/*/readcache_max_filesize
/proc/fs/lustre/osc/*/max_dirty_mb
/proc/fs/lustre/llite/*/max_read_ahead_mb
Lustre - Obest Based File system
GPFS/PanFS - Block based file system.
https://hpc.uni.lu/blog/2016/lustre-tuning/
https://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping
https://www.nics.tennessee.edu/computing-resources/file-systems/io-lustre-tips

Any Linux installation comes along with

1)Installation
2)Configuration
3)Monitoring & tunning
4)Benchmark
5)Monitoring & compare
with old result