As an engineer at Lexis, I would like to share some helpful bits of my experience solving complex business intelligence problems. Recently, I was assigned a rather daunting task of extracting financial information from over half a million documents of financial reports from various companies. This task, though challenging, allowed me to delve deeper into the applications of the more recent AI advances, mainly through OpenAI embeddings and vector databases, for business intelligence problems. How, you ask? Let's take a journey through the process in my comprehensive case study.

## The problem

We were tasked to extract key-value pairs from inconsistent structure and text from business documents. Therefore, we couldn't quickly develop an algorithm to solve the problem and had to implement a solution to leverage the new advances in LLM. We used a cloud solution to train an AI model to extract the key-value pairs. However, it was still pretty challenging to deal with the sheer volume of documents, as not all documents held relevant information, and if we had to process each page, not only would it take more time, but it would be much more expensive. Manually sifting through these documents would be even more time-consuming and economically unviable. The solution? A classification program that combines OpenAI embeddings and ChromaDB vector databases, using Pytesseract for Optical Character Recognition (OCR). 

Before we delve into how these technologies helped me overcome the challenge, let's understand what vector embeddings and vector databases are. 

### Vector embeddings

In a high-dimensional space, vector embeddings are mathematical representations of objects, such as words or documents, capturing semantic meanings based on their context. These vectors provide a way for algorithms to understand the content and context of documents. 

### Vector databases

Vector databases, on the other hand, are databases designed to store and query these vector embeddings efficiently. They enable us to perform similarity search at scale, which is critical in tasks such as semantic search, relatedness search, or, in our example - classification. 

### Implementation

This is where OpenAI and ChromaDB came into the picture. OpenAI provides a powerful tool to generate embeddings for our documents, while ChromaDB allows us to store and query these embeddings efficiently. By leveraging the power of vector embeddings and vector databases, we can classify documents based on how close their vector representations are, which will help us identify relevant pages in a document.

First, I split the PDF documents into individual pages. For each page, I created a MySQL record, applied OCR using Pytesseract, and generated an embedding using OpenAI. Using ChromaDB, I queried the page embedding with the embeddings of already classified pages (that I organized personally) stored in the vector database. The result of the query would give me the most related page to the one I am querying, and I would classify it with the same type, storing the type back in the database.

This process enabled me to filter out irrelevant pages, leaving only the ones that needed further processing. This streamlined the process and reduced the cost by about 80% of the original estimate.

### A basic example

I will give you a simple example to showcase an implementation of a Python program that does a similar job. To implement this solution, you'll need to create a MySQL database that contains the information for each image and a ChromaDB vector database, which you will use to query the vector embeddings.

Here's a simple schema for the MySQL database:

```sql
CREATE TABLE documents (
    id BIGINT UNSIGNED PRIMARY KEY,
    name VARCHAR(255),
    file_path VARCHAR(255), # path of the document in the filesystem
);

CREATE TABLE document_pages (
    id BIGINT UNSIGNED PRIMARY KEY,
    document_id BIGINT UNSIGNED,
    type ENUM(‘invoice’, 'balance_sheet’, ‘income_sheet’, ‘none), # here we store the type of the image once we classify it
    file_path VARCHAR(255),
    FOREIGN KEY (document_id) REFERENCES documents(id)
);
```

We need to manually classify at least a couple hundred images and save them in a folder. The assigned type of each image should be contained in the name of the image, separated by a dash, for example: ```'[name]-invoice.jpg'``` or ```'[name]-none.jpg'```. Then, we will embed these images into a ChromaDB vector database using the following example:

```python
import chromadb
import glob
import os
import pytesseract as pt
from openai import OpenAI
import uuid

openai = OpenAI('your-api-key')

client = chromadb.PersistentClient(path="chromadb")

# create the classification chromadb collection
classification_collection = client.get_or_create_collection(name="classification_collection",  metadata={"hnsw:space": "cosine"})

classified_images = glob.glob("classified_images/*.jpg") # the location of the folder containing the manually classified images

for classified_image in classified_images:
    # get base name of image
    image_type = os.path.basename(classified_image).split(".")[0].split('-')[1]

    # get the ocr text
    ocr_text = pt.image_to_string(classified_image)

    embedding = openai.embeddings.create(
        input=ocr_text.lower(),
        model="text-embedding-ada-002"
    ).data[0].embedding

    # embed the image in the chromadb database
    classification_collection.add(
        embeddings=embedding,
        metadatas={'type': classified_image.type},
        ids = str(uuid.uuid4())
    )

```
After creating the classification database, you can use the following Python code to perform OCR, generate embeddings, and classify the pages of a given document page:

```python
from models import DocumentPage
import chromadb
import pytesseract as pt
from openai import OpenAI

openai = OpenAI('your-api-key')

client = chromadb.PersistentClient(path="chromadb")

# create the classification chromadb collection
classification_collection = client.get__collection(name="classification_collection")

for DocumentPage in DocumentPage.all():
    # get the ocr text
    ocr_text = pt.image_to_string(DocumentPage.file_path)

    embedding = openai.embeddings.create(
        input=ocr_text.lower(),
        model="text-embedding-ada-002"
    ).data[0].embedding

    # embed the image in the chromadb database
    query = classification_collection.query(
        query_embeddings=embedding,
        n_results=1
    )

    DocumentPage.type = query['metadatas'][0][0]['type']

    DocumentPage.save()
```

## Conclusion

This simple example should illustrate the process of classifying images using the latest trends in LLMs and vector databases. The synergy of OpenAI embeddings and ChromaDB vector databases has revolutionized our approach to document classification, making it more efficient and cost-effective. 

The article has given you a glimpse into the power of AI tools in business intelligence. As we continue to explore and experiment, we are excited about the endless possibilities that AI holds for us, and we will keep sharing our insights about how we can use them in real-world scenarios.


Uncover how OpenAI and ChromaDB revolutionized financial data extraction, streamlining document classification for efficiency.

Unleashing the Power of AI in Business Intelligence: Document Classification using OpenAI and ChromaDB

Imagine having a brilliant business idea bubbling with potential, just waiting to become a reality. But there's a catch: you need a developer to make it happen. The problem is the world is brimming with freelancers and development companies, all claiming to be the key to your dreams.  

When it comes to outsourcing, the path ahead can be murky. Hiring a freelancer is a different ballgame than enlisting a software development agency. An agency boasts a powerhouse team of specialized experts with unique skill sets. On the other hand, a freelancer may offer cost savings and flexibility. So, what path should you take when faced with the dilemma of freelancer vs. [software development agency](https://www.lexis.solutions/services/react-native-development)? Your decision today holds the power to shape the future success of your project.

Sorting through these choices can feel overwhelming. But fear not! Let's embark on another of Lexis Solutions' journeys to demystify the selection process.

The agency vs. freelancer quandary may appear deceptively simple, yet its implications must be balanced. Making the correct choice between an agency and a freelancer can transform your final product, for better or worse. So, let's dive headfirst and explore the ideal fit for your project.

## Embrace the freedom

When it comes to freelance developers, they operate in a world of boundless possibilities, offering not just one but three unique modes of engagement. Picture this: they can sign temporary contracts, take freelance gigs, or work remotely full-time. However, like any endeavor, there are risks to be mindful of when collaborating with freelancers, particularly as they juggle multiple projects simultaneously.

### Cost efficiency

The allure of freelance development lies in its ability to provide exceptional value for your investment. Freelancers, armed with nothing more than a trusty computer and an endless supply of caffeine, keep their expenses low and, as a result, offer rates significantly lower than those of software agencies. But remember, cost efficiency doesn't necessarily mean your entire project will come at a reduced price when you hire a freelancer.

### Revel in unparalleled flexibility

Many freelancers embrace working beyond traditional hours and remain responsive to communication. Liberated from the constraints of others' schedules, they can often work with unmatched adaptability, leading to quicker project delivery.

### Harness specialization

Freelancers are masters of their craft, honing their expertise in specific domains. Before bringing a freelancer on board, it's essential to carefully examine their portfolio to understand their specialized skill set. While some freelancers may tout themselves as Jacks-of-all-trades, we all know that true mastery lies in specialization. You can unlock remarkable results by aligning your unique talents with your project's requirements.

### The bad points

While the realm of freelance developers offers enticing advantages, it's crucial to be aware of the potential drawbacks that lurk beneath the surface. Let's shed light on the cons associated with working with freelancers:

### Questionable work quality 

Not all freelancers prioritize the quality of their deliverables. Generic platforms like Upwork need more rigorous vetting processes for their listed freelancers. This means you may encounter individuals solely driven by quick cash, rushing through projects without paying attention to the level of craftsmanship. For such freelancers, reputational risks hold little importance, as they can easily create new accounts or work anonymously.
Heightened risks

Freelancers present inherent risks due to their solitary nature. Even if your chosen freelance developer is highly reliable, unforeseen circumstances can disrupt your project. Without a contingency plan, your project may come to a screeching halt. It's essential to consider the potential consequences and have measures to mitigate those risks.

### Organizational challenges

Working with freelancers can introduce complexities as they juggle multiple clients simultaneously. This increased workload can lead to errors and delays in your product's development. It's crucial to clearly understand your freelancer's capabilities and establish realistic deadlines to minimize the chances of misunderstandings or poor outcomes.

### Vanishing acts

All terms and conditions have been meticulously defined, payments have been made, and the project is ready to take off. Suddenly, your chosen developer disappears into thin air. It may sound like a nightmare, but it's a scenario that can occur if you engage with unreliable freelance platforms or individual contractors lacking the proper means to ensure timely and consistent delivery.

## The triumphs of an agency

A successful outsourcing business typically has a diverse team of experts, including web/mobile app developers, testers, product managers, content creators, designers, analysts, and more. When you hire a development agency, you gain access to a professional support system that can handle all your work efficiently. Their reliability and accountability ensure adherence to work processes and procedures while prioritizing security.
Seamless teamwork

An agency is a pre-established team that has spent years refining its development strategies. While freelancers may not guarantee cooperation, agency employees work together seamlessly daily, fostering unparalleled teamwork and collaboration. Their familiarity with each other's strengths, weaknesses, and work habits enables them to navigate significant challenges effortlessly. No matter how many freelancers you hire, they won't match the level of cooperation an agency guarantees.

When choosing the right IT vendor for your project, consider consulting with top-tier companies through platforms like [Vendorland.com](https://vendorland.com "The voice of the global tech industry through independent research"), which specializes in helping businesses uncover the ideal partners to consult, design, and develop their software needs.

### Extensive expertise

The primary reason agencies cover more ground is their larger workforce. Even the most talented individual can only possess limited knowledge, making an agency more valuable when your project requires diverse areas of expertise. Each employee can specialize in a specific development phase, eliminating the need to search for additional resources. Furthermore, agencies can provide long-term benefits like maintenance, marketing, feature enhancements, and other valuable services by overseeing your product from start to finish.

### Superior quality

Freelancers generally produce high-quality work, but agencies often adhere to even more rigorous quality standards. While freelancers may complete a project without consideration, agencies employ strict review and QA processes. Additionally, freelancers usually need more resources available to agencies for thorough testing. How many independent freelancers have multiple phones for testing purposes?

### Ongoing maintenance & support

Reputable agencies prioritize building long-term customer relationships and upselling additional services. As a result, they are always ready to offer maintenance services, bug fixes, functionality modifications, and other refinements to the products they create or maintain.

### Accurate development estimation

Since most vendors charge hourly, knowing the time required to implement specific functionality by hiring a top software development company can eliminate concerns about unpredictable expenses is crucial. They usually have a well-defined project estimation process, accurately determining the development time required for each feature. And hiring a software development company is an excellent choice for startups and large corporations seeking unique skills, specific timelines, and a reliable contractor committed to project completion.

## Unveiling potential pitfalls

### Beware of hidden costs

When outsourcing software development, be mindful of potential hidden costs. Read the contract the software development company offers. Pay attention to any fine print that may reveal unexpected expenses.

Furthermore, remember that cheaper sometimes doesn’t mean better, especially for complex projects like software development. Opting for offshore developers with lower hourly rates may seem enticing initially, but it could result in higher costs in the long run. Conversely, although some developers may appear expensive initially, they may complete the project more efficiently, ultimately reducing the total cost.

### Increased security and business risks

While reputable software development companies pose minimal risks, hiring the wrong talent can complicate matters. Since software often collects users' data, mishandling or data breaches can have severe consequences. Consider the case of Meta, which was fined $276 million for a data leak impacting millions of Facebook users. Such incidents can be catastrophic for startups and small to medium businesses operating on limited budgets. Additionally, developers are at risk of mishandling sensitive information or intellectual property related to your project.

### Quality control challenges

Failure to properly vet software development companies can lead to partnering with organizations driven primarily by profits rather than a commitment to quality. Conduct thorough due diligence to mitigate this risk by reviewing companies' portfolios and customer feedback.

### Potential for non-delivery

There is always a possibility that the software development company you hire may fail to deliver as promised. They may disappear after collecting upfront costs, struggle to meet your requirements or expectations, provide a flawed product, or become unresponsive. Signing a contract and carefully reviewing its terms beforehand is crucial to protect yourself. Choosing a software development company in a country with a reliable justice system can provide legal recourse if you encounter exploitation.

### Language barriers and cultural differences

Hiring software developers fluent in English and capable of seamless verbal and written communication is advisable. This minimizes the risk of miscommunication and misunderstandings that could negatively impact your working relationship and project outcomes. Additionally, your developers' location in a different time zone may affect decision-making speed and issue resolution. Consider nearshore or onshore developers if this is a concern. Furthermore, be mindful of cultural and political disparities that could hinder effective communication and software development progress, such as cultural differences in values, opinions, and holidays.

## The end of the debate

When hiring a freelancer or a software development agency for your project, opting for a software development agency is a compelling choice. The numerous advantages agencies offer, such as seamless teamwork, extensive expertise, superior quality, long-term maintenance and support, and accurate development estimation, make them a preferred option for many businesses.

But remember, ultimately, the ideal choice depends on your project's requirements, budget, timeline, and risk tolerance. Careful evaluation of the pros and cons, thorough vetting of potential candidates, and clear communication of expectations are essential for making an informed decision that aligns with your project's goals.


Will you tread the freelancer's path of freedom or join forces with the agency's expertise to bring your vision to life?

freelance vs agency

Decoding the Dilemma: Freelancer or Software Development Agency?

## Automation QA's tests before having programming knowledge

## More focus on test design and application understanding

In the world of automation QA, when we first start, and we don't know much about programming, we mainly concentrate on really understanding the application as if we were the customer. This helps us create tests that act just like a regular person using the software, making sure we check it thoroughly in all kinds of situations. Of course, even after we learn programming, our dedication to understanding the application remains. But here's the important part: knowing how to code helps us do an even better job. We can create more advanced tests and make our testing methods more precise and efficient. This way, we keep testing the application like a real user while using our coding skills to do our job correctly.

### Preference for user-friendly automation tools

Additionally, testers with limited programming skills often prefer to stick to user-friendly automation tools with more intuitive interfaces that would require a minimal amount of coding, allowing them to create automated tests more quickly. One advantage here is that they can begin without facing the challenges related to programming languages, which is quite a time saver and lets them dive more into their field in detail.

### Preference for simple testing apps that offer test scripts

QAs who test software without knowing how to program often rely on online testing tools that record and save actions. After a test is registered, it can usually be modified so it can change what you do during different steps and also change the expected result. Such tools copy your interactions and record them in a way that's repeatable and faster than manual work. Later, they can modify them as you please, or you can create your own from existing samples. While they don't write code, we still make the most of what's already available to test software effectively. It's good to note, though, that generally, such online testing tools don't go beyond testing websites only.

### Reliance on regular collaboration with programmers

Given their limited programming expertise, testers often must collaborate more closely with programmers for help. This collaboration becomes essential for addressing script issues and inconsistencies and also handling more advanced automation tasks. 

Working closely with developers can be a valuable learning experience. It ensures that even if you lack programming skills, with enough effort - you can meet development standards and nurture devskills over time. Not to mention that when you start performing your tasks individually - programmers on their own don't have to worry about finding time to fix what you write. They focus on their part, you focus on yours, and productivity increases immeasurably. 

## Benefits of having programming knowledge

### Enhanced test-building skills

Having programming skills equips an automation tester with the ability to build sophisticated and comprehensive test scripts. They can create test scenarios that cover a more comprehensive range of inputs and conditions, resulting in more thorough testing. Additionally, the possession of programming skills allows testers to design tests that are reusable and maintainable, reducing the effort and time in the long run.

### Better debugging process

One aspect of testing, where the benefits of possessing programming skills are sure to make quite an impact, is debugging. Why? Knowing how to read code, analyze your test script errors, and identify issues with your strength saves your time, saves the programmer's time, and last but not least - boosts self-confidence. When a test fails, a QA with the above-mentioned "powers" can trace through the code and successfully pinpoint the problem's root cause and fix it efficiently. Self-education in debugging proves to be quite a utility to master.

Better data handling and validation of the software's behavior
Proficiency in programming enables automation testers to handle and manipulate testing data more effectively. How exactly? For instance, they can create scripts that simulate various users and check if a system differentiates between a new user, a premium user, and an admin. Or let's say they have to test an e-commerce website during which every time a user adds three different products to the cart - the price should be calculated correctly. Possessing skills to perform complex data validations and ensuring that the software behaves as expected under other conditions is an absolute must if we're talking about the reliability of test results.

### More effective collaboration with developers

As we can imagine, programming skills can be the way to "bridge the gap" between an automation QA and the developer for natural reasons, from engaging in relevant product discussions to understanding the code and communicating testing requirements clearly. All these mutual activities signify a more explicit workflow that will result in improved collaboration and communication. And on its own, this will reflect positively on every aspect of the working process.

## How the perspective changes with programming knowledge

### Writing less code that would do more

Acquiring programming knowledge transforms an automation QA's approach to test script development. The QA no longer needs to write long and repetitive code for every single test scenario. Instead, the focus shifts to designing functions and test suites that are somewhat optimized and efficient. Also, this will highlight a level of programming "hygiene" (writing less code while achieving broader test coverage), making the automation process better in the big picture.

### Designing more reusable tests/code

Programming knowledge enables automation QAs to design tests and code that are highly reusable. They can create scripts, functions, constants, and custom commands that can be applied to various test scenarios. It is making the test maintenance easier by far. Such focus on the reusability of the tests accelerates the test development process and proves helpful with fast adaptation of eventual testing of new add-ons to the project.
How programming knowledge boosts the prowess of automation QA

### Enhanced test-building techniques

With some programming skills, we QAs can apply applicable practices to our tests. For example, learning a little bit of HTML, CSS, and DOM tree basics is a good start that can help us to choose better elements to interact with. This means that the more you master them, the more logical assertions, combined with the custom mentioned above commands and costs, you will create. Another example from Cypress is including a well-built beforeEach hook that has saved me from writing the same things for more than 100 tests when I can write it as a precondition only once.

### Test efficiency improves

I've discovered that the power of thinking beyond the basics puts a green flag in your garden. From personal experience, I can tell you that testing a payment system does not include only filling in a card number and clicking the submit button. It's also crucial to not forget to test that you won't be able to pay with a non-active card, an expired one, or an empty one. So far, my programming skills allow me to design and execute tests beyond the standard user flow, ensuring that the payment system is thoroughly and creatively tested. 

### Error handling and reporting improve

Testers with an understanding of programming language can handle errors in their code better. As I already started walking that road, I can confirm that knowing to read your own code is a time saver for both you and the developer. And effective error handling, together with clear error reporting, is essential. A responsible QA also understands the importance of reporting errors that appear in the console during testing, even if they're not immediately visible. As we gain knowledge of programming, we start "speaking the same languages" with developers, so we learn to include the proper terminology like "validation," "verification," etc. 

### Test framework flexibility improves

Knowing more about programming helps us save some time in another way as well. Let's say you have to test a sports betting app on Android and iPhone - the user interface might vary with login buttons, input fields, etc., but that doesn't mean it's a bug. So, instead of writing a long script for two separate tests - you can use coding to adapt the script automatically to the respective system being tested. I already applied it to my approach to work, and it aged pretty well, saving me lots of writing.

## Can we, automation QAs, survive without programming?

Programming skills in automation testing are not a must. But when you want to raise the bar and perform different complex scenarios with your testing tool, you simply need them. Of course, you can always stick to manual work for the more complex tasks. Still, if you put effort and brainwork into learning a little programming and reach a level where no scenario is beyond your expertise to write - that's when you win at testing. Think of programming for automation, like having your own beach house. You can still live without a beach house, but having it kind of changes the horizon, doesn't it? So I suggest you consider building your own!

The Lexis' overview of how an automation QA experiences the testing process without any programming capabilities, why they are essential to acquire, how they change the QA's perspective, and whether testers can make ends meet without them.

pexels-photo-4974922

Programming in Аutomation: Why is it as Еssential as having a Compass in the Wilderness?

clutch-2022

At Lexis Solutions, our commitment to excellence has once again been recognized on a national level. We are proud to announce that for the second consecutive year, we've been honored as one of the top web development companies in Bulgaria by Clutch, a renowned B2B ratings and reviews platform.

![clutch-2024](//images.ctfassets.net/5gydaotbo8jm/59RD0sg8aB4xAh7PhQjJkD/2625ed73950e5a7fc27cdd39364d76d8/top_clutch.co_web_developers_bulgaria_2024.png)

Clutch is widely respected in the industry for its rigorous vetting process, which ensures that only genuine, high-quality reviews make it onto their platform. Their evaluations are rooted in direct client feedback, giving potential customers a clear picture of what it’s like to work with a company like ours.

In their [2024 analysis](http://clutch.co/bg/developers), Clutch has once again highlighted Lexis Solutions as a standout in the Bulgarian development scene. This recognition stems from the consistent praise and high ratings we've received from our clients, a testament to the dedication and skill our team brings to every project.

> "*Earning this accolade two years in a row is an incredible honor for our team. It reinforces our mission to deliver top-tier services that truly make a difference for our clients. We are grateful to everyone who took the time to share their positive experiences on Clutch. This achievement is a shared success between Lexis Solutions and our clients, who inspire us to strive for excellence every day.*"
> 
> Deyan Denchev, Co-Founder of Lexis Solutions.

At Lexis Solutions, we believe that every project is an opportunity to grow and refine our craft. The feedback we receive from our clients not only drives our success but also helps us continually improve and adapt to the ever-changing digital landscape.

As we look forward to the challenges and opportunities 2024 will bring, we remain committed to delivering innovative solutions that meet the evolving needs of our clients. If you have an exciting project in mind, we would love to discuss how we can bring your vision to life.

Clutch's 2024 team article about Lexis Solutions.


Lexis Solutions is a Clutch Top Web Development Company in Bulgaria for 2024

Build your digital solutions with expert help