[{"content":"When working with Kafka, the message you receive isn\u0026rsquo;t always already in the format your application wants to work with.\nIn one of my recent data pipelines, the incoming Kafka payload was an XML string. The final Kafka topic, however, needed data represented using Avro and a schema managed through Confluent Schema Registry.\nThat meant there was a transformation in between:\nKafka ↓ XML String ↓ XML Parsing ↓ Avro SpecificRecord ↓ GenericRecord ↓ Kafka There are a few different technologies involved here—XML, StAX, Avro, GenericRecord, SpecificRecord, and Schema Registry. Individually, none of them is particularly difficult. The interesting part is how they fit together.\nThis article walks through that flow, focusing mainly on how to take an XML payload and build an Avro SpecificRecord from it.\nThe problem with starting from XML XML and Avro represent data differently.\nAn XML payload is essentially a hierarchy of elements:\n\u0026lt;RootObject\u0026gt; \u0026lt;name\u0026gt;...\u0026lt;/name\u0026gt; \u0026lt;resource\u0026gt;...\u0026lt;/resource\u0026gt; \u0026lt;DataField\u0026gt; \u0026lt;name\u0026gt;...\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;...\u0026lt;/value\u0026gt; \u0026lt;/DataField\u0026gt; \u0026lt;DataSet\u0026gt; \u0026lt;name\u0026gt;...\u0026lt;/name\u0026gt; \u0026lt;DataField\u0026gt; \u0026lt;name\u0026gt;...\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;...\u0026lt;/value\u0026gt; \u0026lt;/DataField\u0026gt; \u0026lt;/DataSet\u0026gt; \u0026lt;/RootObject\u0026gt; The actual structure in a production application can obviously be much larger and more deeply nested. The important point is that the XML contains a root object and child objects inside it.\nAvro, on the other hand, works from a defined schema.\nFor the first transformation, I had an Avro schema available locally in the application. The corresponding Java class was generated from that schema, giving me an Avro SpecificRecord and its builder.\nSo rather than trying to construct the final Kafka payload directly from XML, I first converted the XML into a strongly defined Avro object.\nWhy use StAX for the XML parsing? For this transformation, I used the Java XML streaming API:\nXMLInputFactory XMLStreamReader XMLStreamConstants The basic idea is simple.\nInstead of treating the XML as a collection of Java objects immediately, the XMLStreamReader moves through the XML and reports events as it encounters them.\nThe basic pattern looks like this:\nwhile (reader.hasNext()) { int type = reader.next(); if (type == XMLStreamReader.START_ELEMENT) { // process the element } else if (type == XMLStreamReader.END_ELEMENT) { // finish the object when required } } For the implementation, I use the element name to determine what needs to happen next.\nFor example, when the parser encounters a DataSet, I create its builder:\nDataSet.Builder rootSetBuilder = DataSet.newBuilder(); Then its attributes can be populated from the XML:\nrootSetBuilder.setName(...); rootSetBuilder.setResource(...); rootSetBuilder.setOperation(...); Similarly, when a DataField starts, I create a DataField.Builder and populate its values as the corresponding XML elements are encountered.\nThe actual field names in the production XML are business-specific, so the examples here are intentionally simplified.\nReading nested objects This is where the streaming parser becomes a little more interesting.\nThe XML I was processing wasn\u0026rsquo;t just a flat collection of fields. It contained nested objects, including collections of child objects.\nFor example:\nRoot DataSet ├── DataField ├── DataField └── Nested DataSet ├── DataField └── DataField The Avro schema represents these relationships using fields such as arrays.\nWhile parsing the XML, I therefore maintain builders and lists for the objects currently being constructed.\nA simplified version of the pattern is:\nDataSet.Builder rootSetBuilder = null; List\u0026lt;DataSet\u0026gt; dataSets = new ArrayList\u0026lt;\u0026gt;(); List\u0026lt;DataField\u0026gt; dataFields = new ArrayList\u0026lt;\u0026gt;(); DataSet.Builder nestedSetBuilder = null; List\u0026lt;DataField\u0026gt; nestedDataFields = new ArrayList\u0026lt;\u0026gt;(); When the parser encounters a start element, the appropriate builder is created or populated.\nWhen it reaches the corresponding end element, the builder can be completed:\nDataField dataField = dataFieldBuilder.build(); dataFields.add(dataField); For nested objects, the completed child objects are added to the appropriate collection.\nThe parser therefore isn\u0026rsquo;t just reading values. It is keeping track of where those values belong in the resulting Avro object structure.\nIn my implementation, a depth value is also used to distinguish between root-level and nested objects.\nMapping XML values into the SpecificRecord For simple fields, the mapping is fairly direct.\nConceptually:\nXML element ↓ read value ↓ SpecificRecord builder ↓ set field For example:\nif (type == XMLStreamReader.START_ELEMENT) { switch (reader.getLocalName()) { case \u0026#34;value\u0026#34; -\u0026gt; { if (dataFieldBuilder != null) { dataFieldBuilder.setValue( reader.getElementText().trim() ); } } // other elements... } } The important part is that the XML parser knows which builder is currently active.\nAt the end of an object, that builder is built and added to its parent structure.\nFinally, once the XML stream has been completely processed:\nif (rootSetBuilder != null) { rootSetBuilder.setDataFields(dataFields); rootSetBuilder.setNestedDataSets(dataSets); return rootSetBuilder.build(); } The result is the first SpecificRecord.\nSanitizing the XML before parsing There is another step before the XML reaches the stream reader.\nThe Kafka payload arrives as a String, and depending on how the payload is represented, there can be additional wrapping or escaping that needs to be handled before parsing it as XML.\nFor the root payload, my sanitization method does essentially three things:\nprivate String sanitizePayload(String payload) { String s = payload.trim(); if (s.startsWith(\u0026#34;\\\u0026#34;\u0026#34;) \u0026amp;\u0026amp; s.endsWith(\u0026#34;\\\u0026#34;\u0026#34;) \u0026amp;\u0026amp; s.length() \u0026gt; 1) { s = s.substring(1, s.length() - 1); } return StringEscapeUtils.unescapeJava(s); } So the outer whitespace is removed, surrounding quotes are removed when present, and escaped Java characters are unescaped.\nThere is also a separate sanitization method for internal XML data.\nThat method first handles null or blank values, then trims the content. If the value begins with an XML declaration such as:\n\u0026lt;?xml ...?\u0026gt; the declaration is removed before the remaining XML is processed.\nThese are small steps, but they matter because the stream reader needs to receive valid XML rather than the string representation that happened to arrive in the message.\nConfiguring XMLInputFactory The XML reader is created through XMLInputFactory.\nOne part of my configuration looks like this:\nXMLInputFactory factory = XMLInputFactory.newFactory(); factory.setProperty( XMLInputFactory.SUPPORT_DTD, false ); factory.setProperty( XMLInputFactory.IS_COALESCING, true ); factory.setProperty( XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, false ); factory.setProperty( XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, false ); Then the reader is created from the sanitized XML string:\nXMLStreamReader reader = factory.createXMLStreamReader( new StringReader(dataSet) ); The DTD and external-entity related settings are deliberately disabled in this configuration.\nOnce the reader has been created, the application can process the XML event by event.\nWhy not build the GenericRecord immediately? At this point, we have a SpecificRecord.\nBut the final Kafka payload uses a GenericRecord.\nThere is a reason for keeping the intermediate SpecificRecord.\nA SpecificRecord has a generated Java representation based on a known Avro schema. That makes it convenient to build and work with from application code.\nIn my case, the XML-to-object transformation is much easier to express using the generated builders and strongly defined fields.\nThe final structure, however, is handled differently.\nThe target structure is relatively flat, with fields such as strings, timestamps, numbers and decimals at the root level.\nSo after creating the first SpecificRecord, I map its fields into a GenericRecord.\nSpecificRecord to GenericRecord Creating the GenericRecord itself is straightforward.\nThe record is created against the target Avro schema:\nGenericRecord record = new GenericData.Record(schema); Then the fields are populated individually:\nrecord.put( \u0026#34;firstName\u0026#34;, specificRecord.get(\u0026#34;firstName\u0026#34;) ); record.put( \u0026#34;paymentAmount\u0026#34;, specificRecord.get(\u0026#34;paymentAmount\u0026#34;) ); record.put( \u0026#34;paymentCurrency\u0026#34;, specificRecord.get(\u0026#34;paymentCurrency\u0026#34;) ); The actual implementation loops through the required fields and obtains their values from the first SpecificRecord.\nSo the transformation is essentially:\nSpecificRecord │ ├── field A ──→ GenericRecord field A ├── field B ──→ GenericRecord field B ├── field C ──→ GenericRecord field C └── field D ──→ GenericRecord field D There isn\u0026rsquo;t a complicated conversion algorithm here. The important thing is that the GenericRecord is created using the target schema and then populated with values from the SpecificRecord.\nWhere Schema Registry comes in The target Avro schema isn\u0026rsquo;t something the producer invents while publishing the message.\nThe schema for the target record is registered in Confluent Schema Registry.\nThe application retrieves the latest schema metadata using a cached Schema Registry client:\nCachedSchemaRegistryClient client = new CachedSchemaRegistryClient(url, 1000); SchemaMetadata metadata = client.getLatestSchemaMetadata( subject + \u0026#34;-value\u0026#34; ); Schema schema = new Schema.Parser().parse( metadata.getSchema() ); That gives the application an Avro Schema object.\nThat schema is then used when creating the GenericRecord:\nGenericRecord record = new GenericData.Record(schema); This is an important distinction:\nSchema Registry provides the schema. It does not provide the Java SpecificRecord class at runtime.\nThe first SpecificRecord\u0026rsquo;s Java class comes from the Avro schema available in the application. The target schema is retrieved from Schema Registry and used to construct the GenericRecord that will eventually be published.\nPublishing the GenericRecord to Kafka Once the GenericRecord has been built, the final step is publishing it.\nThe application uses Spring Kafka\u0026rsquo;s KafkaTemplate.\nThe value serializer is the Confluent KafkaAvroSerializer:\nkey.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer schema.registry.url=\u0026lt;schema-registry-url\u0026gt; One configuration that is important in this flow is:\nauto.register.schemas=false use.latest.version=true I don\u0026rsquo;t want the producer to automatically register a new schema as part of publishing. The schema is already managed through Schema Registry, so the application is working with the existing schema rather than asking the producer to register a new one as part of the publish operation.\nThe producer also has the Kafka connection, SSL and producer settings configured for the environment, including bootstrap servers, SSL truststore/keystore configuration, acknowledgements, retries, in-flight requests, linger.ms, and compression.\nThe final flow is therefore:\nXML String │ ▼ Sanitize │ ▼ XMLStreamReader │ ▼ SpecificRecord │ ▼ GenericRecord │ │ + target Avro schema ▼ KafkaTemplate │ ▼ KafkaAvroSerializer │ ▼ Kafka SpecificRecord or GenericRecord? This is probably the simplest way I would decide between them based on this implementation.\nSpecificRecord Use it when the application knows the schema and you want a generated Java representation to work with.\nIn this pipeline, that makes the XML transformation easier because the application can work with generated builders and defined fields.\nGenericRecord Use it when the record needs to be handled more dynamically against an Avro Schema.\nIn this pipeline, the GenericRecord is used at the final publishing stage, where the target schema comes from Schema Registry.\nThe distinction isn\u0026rsquo;t that one is universally better than the other. They solve slightly different problems.\nIn this particular flow, using both gives a fairly clean separation:\nXML │ ▼ SpecificRecord │ │ easier application-side construction ▼ GenericRecord │ │ target schema ▼ Kafka The complete picture Putting everything together, the transformation looks like this:\nConfluent Schema Registry │ │ latest target schema ▼ Kafka XML ──→ Sanitize ──→ StAX Parser ──→ SpecificRecord │ │ field mapping ▼ GenericRecord │ │ target schema ▼ KafkaTemplate │ ▼ Kafka The XML side and Avro side are doing different jobs.\nThe XML parser is responsible for understanding the incoming hierarchical structure.\nThe SpecificRecord gives the application a strongly defined Java representation to work with.\nThe GenericRecord provides the final record representation against the target schema.\nAnd Schema Registry provides the schema that defines that target structure.\nOnce those responsibilities are separated, the transformation becomes much easier to reason about.\nFinal thoughts XML and Avro can initially feel like two completely different worlds. One is hierarchical and text-based, while the other is schema-driven and designed for structured serialization.\nThe important part is not trying to make one directly behave like the other.\nInstead, treat the transformation as a series of small steps:\nclean the input → read the XML → build the SpecificRecord → map the required fields → create the GenericRecord with the target schema → publish it through Kafka.\nFor me, the most useful part of this approach is that the XML parsing logic doesn\u0026rsquo;t have to know anything about Kafka serialization, and the Kafka producer doesn\u0026rsquo;t have to know how the original XML was structured.\nEach step has one job.\nAnd that makes the whole pipeline much easier to work with.\n","permalink":"https://yshashanky.github.io/blog/xml-to-avro-specificrecord/","summary":"\u003cp\u003eWhen working with Kafka, the message you receive isn\u0026rsquo;t always already in the format your application wants to work with.\u003c/p\u003e\n\u003cp\u003eIn one of my recent data pipelines, the incoming Kafka payload was an XML string. The final Kafka topic, however, needed data represented using Avro and a schema managed through Confluent Schema Registry.\u003c/p\u003e\n\u003cp\u003eThat meant there was a transformation in between:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eKafka\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  ↓\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eXML String\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  ↓\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eXML Parsing\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  ↓\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eAvro SpecificRecord\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  ↓\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eGenericRecord\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  ↓\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eKafka\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThere are a few different technologies involved here—XML, StAX, Avro, GenericRecord, SpecificRecord, and Schema Registry. Individually, none of them is particularly difficult. The interesting part is how they fit together.\u003c/p\u003e","title":"From XML to Avro: Building SpecificRecord and Publishing a GenericRecord with Kafka"},{"content":"When I first looked at the requirement for one of our Kafka pipelines, using KStream felt like the obvious choice.\nThe requirement itself was simple: consume a message from one Kafka topic, validate and process it, and publish the result to another Kafka topic.\nThe part that made it less simple was the infrastructure. The source and destination topics were on two completely separate Kafka clusters. Those clusters also had different security configurations.\nThat difference ended up changing the architecture.\nThe requirement The pipeline was processing transaction-related records for a banking use case, where reliability and data consistency were important.\nThe expected flow was straightforward:\nConsume a message. Validate and process it. Transform it into the required output. Publish it to the destination topic. The pipeline was expected to handle around 80,000 messages per hour.\nAt first, I thought Kafka Streams would fit this very well. A stream comes in, some processing happens, and another stream goes out. That is exactly the kind of flow Kafka Streams is designed to make convenient.\nSo we started with KStream.\nWhere things became interesting The source and destination were not just two topics with different names.\nThey belonged to different Kafka clusters. The consumer side had one set of connection and security requirements. The producer side had another.\nFor example, the source cluster used SASL + SSL, while the destination cluster used SSL. They also had their own bootstrap servers, certificates, keystores, truststores and related credentials.\nWe configured the Streams application and started testing.\nThe interesting part was that the consumer side was working. Messages were being consumed and the processing logic was running. But the producer was repeatedly failing to connect to the destination cluster.\nThe error we kept seeing was a connection timeout. It would retry the connection, but the result was the same connection timeout again.\nWe spent roughly three to four hours trying to get that approach working. One of the things that helped us narrow it down was testing the same kind of flow with topics from the same cluster. That worked.\nOnce the source and destination were on different clusters with their different configurations, we hit the problem again.\nThat gave us a much stronger signal that this wasn\u0026rsquo;t simply a bad topic configuration or a temporary connectivity problem.\nWe went back through the Kafka Streams configuration model and some previous implementations/documentation. The issue was that the conventional Streams setup was not a good fit for what we were trying to do: independently connect the consumer side to one Kafka cluster and the producer side to another cluster with separate infrastructure and security configurations.\nAt that point, continuing to fight the configuration didn\u0026rsquo;t make much sense.\nWe already had the answer we needed.\nThe annoying part: we had already done the work This was probably the most frustrating part of the whole thing.\nWe had already spent time implementing the KStream-based solution. The problem was not that the processing logic was wrong. The problem was that the architecture we had chosen didn\u0026rsquo;t fit the infrastructure we actually had.\nChanging the approach meant rewriting the code and going through the architecture approval process again. We were also working against a tight delivery timeline, so we had to put in some additional effort to complete the work on time.\nIt wasn\u0026rsquo;t a huge schedule impact in the end, but it was avoidable rework.\nIf we had validated the Kafka cluster boundary before starting the implementation, we could have chosen the second approach from the beginning.\nSwitching to a separate Consumer and Producer The replacement was much more explicit.\nInstead of using one Kafka Streams topology, we used a Spring Kafka consumer and a separate producer configuration in the same Spring Boot application.\nThe flow became:\nKafka Cluster A | | @KafkaListener v Consume | v Validate / Process / Transform | v KafkaTemplate | v Kafka Cluster B The important difference was that the consumer and producer were now independent clients.\nThe consumer had its own configuration for Cluster A.\nThe producer had its own configuration for Cluster B.\nThat made the separation much easier to reason about.\nThe listener itself was intentionally simple. It received a ConsumerRecord, passed it to the processing service, and worked with a CompletableFuture representing the asynchronous processing.\nA simplified version of the listener looked like this:\n@KafkaListener( topics = \u0026#34;${kafka.consumer.topic}\u0026#34;, groupId = \u0026#34;${spring.kafka.consumer.group-id}\u0026#34;, containerFactory = \u0026#34;kafkaListenerContainerFactory\u0026#34; ) public void consumeMessage( ConsumerRecord\u0026lt;String, String\u0026gt; message, Acknowledgment acknowledgment) { CompletableFuture\u0026lt;Void\u0026gt; processingFuture = asyncService.process(message); processingFuture.whenComplete((ignored, ex) -\u0026gt; { if (ex == null) { acknowledgment.acknowledge(); } else { // Do not acknowledge the message } }); } The actual implementation contains logging and additional error handling, but this is the important part of the flow.\nWe used manual acknowledgment. The message was not acknowledged immediately after the listener received it. We waited for the asynchronous processing to complete successfully.\nOnly then did we acknowledge it, after which Spring handled the corresponding offset commit.\nThat distinction mattered because publishing the message successfully was part of completing the processing.\nThe producer was completely separate For the producer, we used a dedicated KafkaTemplate\u0026lt;String, GenericRecord\u0026gt;.\nThe producer configuration had its own bootstrap servers and security properties for the destination cluster.\nSome of the relevant configuration included:\nprops.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, producerBootstrapServers); props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, producerSecurityProtocol); props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, truststoreLocation); props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, truststorePassword); props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, keystoreLocation); props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, keystorePassword); There were also producer settings for acknowledgments, retries, in-flight requests, linger and compression.\nThe producer was responsible for publishing the transformed GenericRecord to the appropriate destination topic.\nThat separation made debugging much easier too.\nIf the consumer had a problem, we could look at the Cluster A configuration.\nIf the producer had a problem, we could look at the Cluster B configuration.\nThere was less ambiguity about which client was responsible for which connection.\nWhat happens when publishing fails? This was an important part of the implementation.\nWe didn\u0026rsquo;t want to acknowledge the source message just because our application had successfully started processing it.\nThe producer send was asynchronous.\nConceptually, the processing looked like:\nreturn kafkaTemplate .send(publishTopic, consumerRecord.key(), payment) .thenAccept(result -\u0026gt; { // Publish succeeded }) .handle((result, ex) -\u0026gt; { if (ex == null) { return CompletableFuture.completedFuture(null); } // Handle publish failure return handlePublishFailure(...); }) .thenCompose(Function.identity()); The actual implementation also handled the transformation and topic selection before the send. For example, the processed transaction could be routed to one of the appropriate destination topics based on the resulting transaction state.\nThe important point is that successful publishing and source acknowledgment were connected.\nIf publishing completed successfully, the processing future completed successfully and the listener acknowledged the source message.\nIf publishing failed, the processing future did not complete successfully, so the source message was not acknowledged.\nRetry vs DLQ We also didn\u0026rsquo;t treat every failure as a retryable failure.\nFor producer/connectivity-related failures, we used retries.\nThe sequence was:\nInitial publish | X | wait 1s | Retry 1 | X | wait 3s | Retry 2 | X | wait 5s | Retry 3 | X | DLQ So there was an initial attempt followed by three retries.\nThe retry delays were:\nFirst retry: 1 second Second retry: 3 seconds Third retry: 5 seconds The retries were primarily for producer/connectivity-related failures.\nWe didn\u0026rsquo;t want to keep retrying a message when the message itself was the problem.\nFor example, if the data was corrupt, required fields were missing, a value was null where it shouldn\u0026rsquo;t be, or the data type was incorrect, retrying the same message wasn\u0026rsquo;t going to fix it.\nThose cases went directly to the DLQ.\nThere is an important limitation here There is a subtle point about acknowledgments and offsets that is easy to miss.\nThe source offset is acknowledged only after successful processing/publishing.\nThat gives us a useful guarantee: if the application fails before the source offset is acknowledged, the source record can be processed again.\nBut this does not give us exactly-once processing across two independent Kafka clusters.\nConsider this sequence:\n1. Consume message from Cluster A 2. Publish successfully to Cluster B 3. Application fails 4. Source offset was not committed 5. Message is consumed again 6. It may be published to Cluster B again So there is a possibility of duplicate publication if the destination publish succeeds but the source offset is not committed before the application fails.\nThat is an important distinction. The design gives us a practical at-least-once style processing behavior, but we should not describe it as exactly-once across the two clusters.\nFor this pipeline, that tradeoff was understood and handled as part of the design.\nWhat I would choose today The main thing I took away from this was not that KStream is bad.\nIt isn\u0026rsquo;t.\nIf the consumer and producer are working against the same Kafka cluster and the same general configuration, I would still prefer KStream for this kind of processing. It simplifies the implementation considerably. The consume-process-publish flow maps naturally to a stream topology, and there is less client configuration to manage directly.\nThe decision changes when the infrastructure changes.\nIf the source and destination are in different Kafka clusters and need independent connection and security configurations, I would choose a separate consumer + producer approach. In our case, that meant using @KafkaListener for the source and KafkaTemplate for the destination.\nThe lesson for me was fairly simple: don\u0026rsquo;t choose the abstraction before checking the infrastructure boundary.\nWe already knew that the source and destination were in different clusters. What we didn\u0026rsquo;t account for initially was what that meant for the Kafka Streams approach.\nIf I were starting the same pipeline today, I\u0026rsquo;d validate that first.\nThat would probably save the three or four hours of debugging — and more importantly, the code rewrite and second architecture approval that followed it.\nThat’s the whole story. Onward to the next surprise.\n","permalink":"https://yshashanky.github.io/blog/kstream-vs-consumer-producer/","summary":"\u003cp\u003eWhen I first looked at the requirement for one of our Kafka pipelines, using KStream felt like the obvious choice.\u003c/p\u003e\n\u003cp\u003eThe requirement itself was simple: consume a message from one Kafka topic, validate and process it, and publish the result to another Kafka topic.\u003c/p\u003e\n\u003cp\u003eThe part that made it less simple was the infrastructure. The source and destination topics were on two completely separate Kafka clusters. Those clusters also had different security configurations.\u003c/p\u003e","title":"KStream vs Consumer-Producer: Choosing the Right Architecture"},{"content":"Came across this algorithm while exploring how to randomize data in an array or list, and it is working quite efficiently and accurately. It is also known as the Knuth shuffle, developed by Ronald Fisher and Frank Yates in 1938.\nHow it works This algorithm works by iterating over the elements of the array in reverse order and swapping each element with a randomly selected element that comes before it in the array. To find the random index, I am multiplying the index by the output of the Math.random() function. This process ensures that each element has an equal probability of ending up in any position in the shuffled array.\nI am using this to get a subset of the random array. Below is a simple implementation of the Fisher-Yates algorithm in JavaScript:\nfunction getRandomSubset(arr, length) { const copyArray = arr.slice(); // Shuffle the array using the Fisher-Yates algorithm for (let i = copyArray.length - 1; i \u0026gt; 0; i--) { const j = Math.floor(Math.random() * (i + 1)); [copyArray[i], copyArray[j]] = [copyArray[j], copyArray[i]]; } return copyArray.slice(0, length); } Walkthrough In this implementation:\nWe start from the last element of the array and iterate backward (let i = copyArray.length - 1; i \u0026gt; 0; i--). For each element at index i, we generate a random index j using the function Math.floor(Math.random() * (i + 1)) such that 0 \u0026lt;= j \u0026lt;= i. We then swap the elements at indices i and j. This process continues until we reach the first element of the array, resulting in a shuffled array. Additionally, I am slicing the random array to the required length. I hope it helps, and that\u0026rsquo;s all for today. Until next time, take care and have a great time!\n","permalink":"https://yshashanky.github.io/blog/fisher-yates-algorithm/","summary":"\u003cp\u003eCame across this algorithm while exploring how to randomize data in an array or list, and it is working quite efficiently and accurately. It is also known as the Knuth shuffle, developed by Ronald Fisher and Frank Yates in 1938.\u003c/p\u003e\n\u003ch3 id=\"how-it-works\"\u003eHow it works\u003c/h3\u003e\n\u003cp\u003eThis algorithm works by iterating over the elements of the array in reverse order and swapping each element with a randomly selected element that comes before it in the array. To find the random index, I am multiplying the index by the output of the \u003ccode\u003eMath.random()\u003c/code\u003e function. This process ensures that each element has an equal probability of ending up in any position in the shuffled array.\u003c/p\u003e","title":"Fisher-Yates Algorithm"},{"content":"In the last article, we discussed a few introductory topics that will be helpful to get started with k6. Today, we are going to discuss how to write and structure the tests.\nHow to structure tests? You can either write everything in a single file or create multiple files depending on the number of microservices. Here\u0026rsquo;s how I do it:\nCreate a main.js file at the root of the project folder. This file should include running configurations, report imports and methods, setup functionality, constants, and the main function. Keep the rebuilt k6 executable file at the root of the project folder; you can ignore it during the testing process. Create a folder for microservices, named controllers at the root of the project folder. This folder will contain multiple .js files for different microservices, depending on how you have separated them. Each .js file inside the controllers\u0026rsquo; folder can contain a single or multiple microservices. Create a folder for reports at the root of the project. Create a folder to store all the required test data. This is pretty much everything you need to do.\nWriting the tests k6 only supports JavaScript as of now. Let\u0026rsquo;s start with main.js. Sharing a sample below:\n// importing all the required modules from k6 import { check, fail, group, sleep } from \u0026#34;k6\u0026#34;; import http from \u0026#34;k6/http\u0026#34;; // import the required files for generating PDF reports import { textSummary } from \u0026#34;https://jslib.k6.io/k6-summary/0.0.1/index.js\u0026#34;; import { htmlReport } from \u0026#34;https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js\u0026#34;; // import controllers import { getMenu } from \u0026#34;./controllers/menu.js\u0026#34;; import { getGuestProfile, searchGuestProfile, } from \u0026#34;./controllers/guestProfile.js\u0026#34;; // Set environment const env = \u0026#34;xs\u0026#34;; // Set other constants const returnReport = true; const baseUrl = `https://test.${env}.com`; const authUrl = `https://test-auth.${env}.com`; export const options = { // Runs the load test with the same number of VUs for a specific duration vus: 750, duration: \u0026#34;15m\u0026#34;, // Runs the load test in stages with a different number of Virtual Users (VUs) and durations // stages: [ // { duration: \u0026#34;20s\u0026#34;, target: 10 }, // { duration: \u0026#34;30s\u0026#34;, target: 100 }, // { duration: \u0026#34;3m\u0026#34;, target: 200 }, // ], // Checks and thresholds for the load test thresholds: { // Default checks checks: [\u0026#34;rate\u0026gt;0.9\u0026#34;], // Success and failed request counts and checks // failedGetMenuRequestCount: [\u0026#34;count \u0026lt;= 10\u0026#34;], // successGetMenuRequestCount: [\u0026#34;count \u0026gt;= 10\u0026#34;], // failedGetGuestProfileRequestCount: [\u0026#34;count \u0026lt;= 10\u0026#34;], // successGetGuestProfileRequestCount: [\u0026#34;count \u0026gt;= 10\u0026#34;], // failedSearchGuestProfileRequestCount: [\u0026#34;count \u0026lt;= 10\u0026#34;], // successSearchGuestProfileRequestCount: [\u0026#34;count \u0026gt;= 10\u0026#34;], // Overall API timings and checks apiTimings_getMenu: [\u0026#34;p(95) \u0026lt; 2000\u0026#34;], apiTimings_getGuestProfile: [\u0026#34;p(95) \u0026lt; 2000\u0026#34;], apiTimings_searchGuestProfile: [\u0026#34;p(95) \u0026lt; 2000\u0026#34;], }, }; // Function to do setup befor starting load test export function setup() { const res = getToken(); if (res.status !== 200) fail(\u0026#34;Failed to get the auth token\u0026#34;); const token = JSON.parse(res.body).access_token; return { token }; } // Function to generate token export function getToken() { const headers = { Authorization: \u0026#34;Basic test\u0026#34;, \u0026#34;Content-Type\u0026#34;: \u0026#34;application/x-www-form-urlencoded\u0026#34;, \u0026#34;Cache-Control\u0026#34;: \u0026#34;no-cache\u0026#34;, }; const payload = { grant_type: \u0026#34;access\u0026#34;, scope: \u0026#34;roles\u0026#34;, }; const url = `${authUrl}/oauth2/access_token`; const res = http.post(url, payload, { headers: headers }); return res; } // This function returns summary and generate reports export function handleSummary(data) { const stdout = textSummary(data, { indent: \u0026#34; \u0026#34;, enableColors: returnReport, }); const testResult = `reports/${env}testResult${ new Date().toISOString().replace(/[:]/g, \u0026#34;_\u0026#34;).split(\u0026#34;.\u0026#34;)[0] }.html`; return returnReport ? { [testResult]: htmlReport(data), // \u0026#34;reports/testResult.json\u0026#34;: JSON.stringify(data, null, 2), //Uncomment this line if you require the result in a JSON file. stdout, } : { stdout, }; } // main function export default function (data) { const res = http.get(`${baseUrl}/actuator/health`); check(res, { appIsHealthy: (res) =\u0026gt; JSON.parse(res.body).status === \u0026#34;UP\u0026#34;, }); group(\u0026#34;Menu\u0026#34;, () =\u0026gt; { getMenu(baseUrl, data.token); //dependent on meal period }); group(\u0026#34;Guest profile\u0026#34;, () =\u0026gt; { getGuestProfile(baseUrl, data.token); searchGuestProfile(baseUrl, data.token); }); sleep(1); } Let\u0026rsquo;s discuss a few things done in the above code:\nStart by importing all the required modules from k6. Then import any external dependencies that are required. Next, import the microservices created in the controllers, which will essentially be functions as well. Set the constants such as environment variables, API URLs, and any other necessary details. Define the const options, which is a default from k6. It will contain the running configurations, checks, thresholds, and any other required custom details. There are two ways to run the test: one is in stages and the other is with a constant number of virtual users. Both options are shared above; choose as per your needs. You can add as many stages as you want. There are no restrictions, just make sure you have enough resources to run them. Then, there\u0026rsquo;s the setup function, which runs before starting the main function. It\u0026rsquo;s a good point to obtain access tokens and other details needed for the tests. After that, we have the handleSummary function, which is responsible for storing the test reports. In this function, a small function is added to get the report name with a timestamp, which helps when sorting through multiple reports. Finally, there\u0026rsquo;s the main function, responsible for running the complete tests. Within this function, the code checks for the service\u0026rsquo;s health before making calls to different microservices. It\u0026rsquo;s important to verify if the service is reachable before making requests, as the purpose of the test is to determine how much load it can handle. If the service is reachable, it proceeds to make calls to the microservices and updates the report. Sleep functions are used to wait for a second before making the next request once the previous request is completed. This is pretty much everything about main.js. If you need more information about any function, you can refer to the official docs.\nNow let\u0026rsquo;s take a look into the microservice file. I\u0026rsquo;m sharing a sample below:\n// importing all the required modules import { URLSearchParams } from \u0026#34;../dependencies/URLIndex.js\u0026#34;; import { check } from \u0026#34;k6\u0026#34;; import http from \u0026#34;k6/http\u0026#34;; import { Trend, Counter } from \u0026#34;k6/metrics\u0026#34;; import { getvenueId } from \u0026#34;../testdata/readData.js\u0026#34;; // Custom metrics const failedRequestCount = new Counter(\u0026#34;failedGetMenuRequestCount\u0026#34;); const successRequestCount = new Counter(\u0026#34;successGetMenuRequestCount\u0026#34;); const getTrend = new Trend(\u0026#34;apiTimings_getMenu\u0026#34;); // Method to send getMenu requests export function getMenu(baseUrl, token) { const searchParams = new URLSearchParams([ [\u0026#34;place\u0026#34;, `${getvenueId()}`], [\u0026#34;category\u0026#34;, \u0026#34;food\u0026#34;], ]); const res = http.get(`${baseUrl}/menu?${searchParams}`, { headers: { Authorization: `Bearer ${token}`, Accept: \u0026#34;/\u0026#34;, }, }); const result = check(res, { getMenu: (res) =\u0026gt; res.status === 200, }); failedRequestCount.add(!result); successRequestCount.add(result); getTrend.add(res.timings.duration); } Let\u0026rsquo;s discuss a few things done in the above code:\nStart by importing all the required modules. Create custom metrics for each API. By default, the final result will contain consolidated stats of all the APIs, but if you need to see the stats of each API separately, you can add similar custom metrics. Create a function for each API. It will make the call, check if the returned response is successful or not, and then update the custom metrics accordingly. That\u0026rsquo;s all you need to do for other microservices as well, and you are good to go.\nSimilarly, you can add more APIs and then just import them into the main.js and add them to the main function. No need to change any configuration. This will make your test template easily extensible.\nI hope it helps, and that\u0026rsquo;s all for today. Next time, we\u0026rsquo;ll talk about how to randomize and access the data. Until then, have a great time!\n","permalink":"https://yshashanky.github.io/blog/why-and-how-to-use-k6-part-2/","summary":"\u003cp\u003eIn the last \u003ca href=\"/blog/why-and-how-to-use-k6/\"\u003earticle\u003c/a\u003e, we discussed a few introductory topics that will be helpful to get started with k6. Today, we are going to discuss how to write and structure the tests.\u003c/p\u003e\n\u003ch3 id=\"how-to-structure-tests\"\u003eHow to structure tests?\u003c/h3\u003e\n\u003cp\u003eYou can either write everything in a single file or create multiple files depending on the number of microservices. Here\u0026rsquo;s how I do it:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCreate a \u003ccode\u003emain.js\u003c/code\u003e file at the root of the project folder. This file should include running configurations, report imports and methods, setup functionality, constants, and the main function.\u003c/li\u003e\n\u003cli\u003eKeep the rebuilt k6 executable file at the root of the project folder; you can ignore it during the testing process.\u003c/li\u003e\n\u003cli\u003eCreate a folder for microservices, named \u003ccode\u003econtrollers\u003c/code\u003e at the root of the project folder. This folder will contain multiple \u003ccode\u003e.js\u003c/code\u003e files for different microservices, depending on how you have separated them.\u003c/li\u003e\n\u003cli\u003eEach \u003ccode\u003e.js\u003c/code\u003e file inside the controllers\u0026rsquo; folder can contain a single or multiple microservices.\u003c/li\u003e\n\u003cli\u003eCreate a folder for reports at the root of the project.\u003c/li\u003e\n\u003cli\u003eCreate a folder to store all the required test data.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis is pretty much everything you need to do.\u003c/p\u003e","title":"Why and How to Use It - Part II"},{"content":"k6 is currently one of the best and easiest-to-set-up load testing frameworks. There are tons of articles and guides on how to get started with k6 for load testing, but after a point, they aren\u0026rsquo;t much help when the complexity starts increasing. In this article, I am going to discuss a few things that saved me a lot of time. You\u0026rsquo;re probably familiar with the next few sections, but let\u0026rsquo;s go over them once more.\nWhat is k6? Grafana k6 is an open-source load testing tool that makes performance testing easy and productive for engineering teams. k6 is free, developer-centric, and extensible. You can use Grafana Cloud with k6 if you want to save resources and time, as it will configure and handle multiple things for you. However, Grafana Cloud is paid.\nk6 is developed by Grafana Labs and the community.\nKey features k6 is packed with features, which you can learn all about in the documentation. Key features include:\nCLI tool with developer-friendly APIs. Scripting in JavaScript ES2015/ES6 - with support for local and remote modules Checks and Thresholds - for goal-oriented, automation-friendly load testing Read more about it from the official doc.\nGood to know: You don\u0026rsquo;t need to install k6 if you are using VS Code. You can use the executable file directly to run the tests from the terminal.\nWhat will be included in the results output? The results generated by k6 are quite descriptive. For quick analysis, you can check the summary metrics displayed in the terminal output. For more in-depth analysis, granular time-series data is available. Both metrics and result summaries can be customized as needed. Outputs can be saved in JSON or CSV format, or any other built-in output format.\nRead more about it from the official doc.\nThe best way to save and share output is in PDF format, but officially, it is only available on annual Pro and Enterprise plans. However, there is an alternative solution: you can use k6-reporter. It is one of the best and easiest to set up; you just need to add two imports to your tests and you are done. If you have basic knowledge of EJS and JS, you can tailor it to your needs.\nThis is all you need to do:\n// This will export to HTML as filename \u0026#34;result.html\u0026#34; AND also stdout using the text summary import { htmlReport } from \u0026#34;https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js\u0026#34;; import { textSummary } from \u0026#34;https://jslib.k6.io/k6-summary/0.0.1/index.js\u0026#34;; export function handleSummary(data) { return { \u0026#34;result.html\u0026#34;: htmlReport(data), stdout: textSummary(data, { indent: \u0026#34; \u0026#34;, enableColors: true }), }; } In many cases, running a test for a long period can be time-consuming, especially if waiting for the results. To address this, you can stream live test results using various extensions provided by k6. One such extension is the xk6-dashboard, which, although a bit tricky to set up initially, becomes straightforward afterward.\nThere are two methods to use it: either by utilizing pre-built k6 binaries from the Releases page, or by rebuilding the installed k6 executable file with this extension. The latter option may require a few extra steps but offers improved usage. The choice is yours. I opted for the second option and generated a new executable. Below, I\u0026rsquo;ll share the command used and some info on the parameters I\u0026rsquo;m currently using.\n.\\k6.exe run --out \u0026#39;dashboard=period=10s\u0026amp;report=.\\reports\\loadTestCustomReport.html\u0026#39; main.js Period: This parameter defines how frequently the live tests should be updated. For tests that run for several hours, it\u0026rsquo;s advisable to set it around 20 seconds. However, for shorter tests, a period of around 10 seconds is recommended for better analysis. Report: If you provide the path where your results need to be stored, they will be saved in an HTML file. This file allows you to share and review the details of each second of the test. The best approach would be to add an extension for PDF output while using the dashboard. The PDF can serve as a summary, while HTML can be considered for deep analysis.\nTo make the best use and avoid typing lengthy commands repeatedly, create a build file. This will make it easier for non-technical team leads or anyone to run a quick test and obtain the results. A sample is shared below:\n{ \u0026#34;version\u0026#34;: \u0026#34;2.0.0\u0026#34;, \u0026#34;tasks\u0026#34;: [ { \u0026#34;label\u0026#34;: \u0026#34;Run k6 Load Test\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;command\u0026#34;: \u0026#34;.\\\\k6.exe\u0026#34;, \u0026#34;args\u0026#34;: [ \u0026#34;run\u0026#34;, \u0026#34;--out\u0026#34;, \u0026#34;\u0026#39;dashboard=period=10s\u0026amp;report=.\\\\reports\\\\loadTestCustomReport.html\u0026#39;\u0026#34;, \u0026#34;main.js\u0026#34; ], \u0026#34;group\u0026#34;: { \u0026#34;kind\u0026#34;: \u0026#34;build\u0026#34;, \u0026#34;isDefault\u0026#34;: true }, \u0026#34;presentation\u0026#34;: { \u0026#34;reveal\u0026#34;: \u0026#34;always\u0026#34;, \u0026#34;panel\u0026#34;: \u0026#34;dedicated\u0026#34; } } ] } Ending this article here. In the next one, we will be talking about how to structure and write your tests. Thank you for your time.\n","permalink":"https://yshashanky.github.io/blog/why-and-how-to-use-k6-part-1/","summary":"\u003cp\u003ek6 is currently one of the best and easiest-to-set-up load testing frameworks. There are tons of articles and guides on how to get started with k6 for load testing, but after a point, they aren\u0026rsquo;t much help when the complexity starts increasing. In this article, I am going to discuss a few things that saved me a lot of time. You\u0026rsquo;re probably familiar with the next few sections, but let\u0026rsquo;s go over them once more.\u003c/p\u003e","title":"Why and How to Use It - Part I"}]