FREE LOOK

Ubuntu Find Out Your Block Size

The block size you have on your drives can significantly affect performance, but can also cause a lot of wasted space if you set it too large. In simple terms, the block size represents the minimum amount of physical disk space that is taken up by a file. Hence, if you have lots of tiny text files, then its a good idea to have a small block size in order to reduce wasted space. However, if you are using HDDs (traditional hard drives) to just store large media files such as movies and music, then you may find that you want to set a larger block size of perhaps 1 MiB for better read/write performance.

Why would Increasing The Block Size Increase Performance?

Each block needs to have some assosciated metadata. Thus if you have a smaller block size, then there is more metadata that has to be written when you write the entire file (e.g. file transfers). I assume this is particularly relevant to filesystem level encryption (e.g. truecrypt) where a tiny change to a file within the volume results in all the bits of the volume (thus all the blocks) changing and having to be updated. This is why serverbear benchmarks drives with block sizes at 1M and 64k as shown below:

Whats the Command to Find Out My Block Size?

sudo dumpe2fs /dev/sda1 | fgrep -e Block size

References

• Ubuntu Forums - command to get block size in Ubuntu?

download file now

Trident What is the batch size for KafkaSpout

Today i encountered a problem in which i need to know what is the batch size for a TransactionalTridentKafkaSpout.

The default output field by KafkaSpout is "str", which can be of variable length. By some simple thought, it is reasonable that kafka spout will set the batch size dynamically based on the actual sizes of the "str" messages it retrieves in a batch. In other words, i suspect that the batch size may be determined by the following pseudo codes:

batchSize = 0;

totalBytesConsumed=0;

while(totalBytesConsumed < maxBytesConsumed)
{
    totalBytesConsumed+= getMessageSize("str");
    batchSize ++;

}

After quickly read through trident-kafkas source codes on TransactionalTridentKafkaSpout and trace down to the following classes and their methods:

TransactionalTridentKafkaSpout.getEmitter(...)
TridentKafkaEmitter.fetchMessages(...)
KafkaUtils.fetchMessages(...)

by the builder.addFetch(...) line in "KafkaUtils.fetchMessages(...)", it looks like the batch size is determine by a variable defined in KafkaConfig.fetchSizeBytes, which is defaulted to 1024 * 1024.

I also noticed another variable KafkaConfig.bufferSizeBytes, which is used by the SimpleConsumer class, this is also defaulted to 1024 * 1024.

Therefore i suspect that the batch size of the kafka spout depends on both Math.Min(KafkaConfig.bufferSizeBytes, kafkaConfig.fetchSizeBytes).

After some googling, I noticed for  handling huge data in Kafka, the following settings have been used:

kafkaConfig.bufferSizeBytes = 1024 * 1024 * 4;
kafkaConfig.fetchSizeBytes = 1024 * 1024 * 4;

download file now