February 2024 Update

After the busy last months that resulted in almost an overall make-over of the platform, in February, we've focused on stability improvements across the board. We've also made a number of improvements to the dashboard UI, the evaluations, and the templates.

Improved UI for Traces

We've improved the UI of the traces to be more readable and work better with smaller screens.

Evaluation SDK in general availability

All Unlimited and Enterprise users now have access to the Evaluation SDK.

Create CI pipelines for your agents, benchmark your RAG pipelines, and more. The SDK is now generally available and ready for production use.

Here is an example of how to use the SDK:

testing_dataset = lunary.get_dataset("test-cases")

for item in testing_dataset:

  # Run your agent on each input of the dataset
  res = support_agent(item.input)

  # Evaluate the output
  passed, details = lunary.evaluate(
    checklist='ci-checks',
    input=item.input,
    output=res,
    ideal_output=item.ideal_output
  )

  if passed:
      print(f"Test passed!", 'green')
  else:
      print(f"Test failed!")

Text-only datasets

You can now use text-only datasets (instead of chat-messages datasets only before). This makes it easier to create test cases for your custom agents.

get_langchain_template methods

For those that work with LangChain, we've added new methods to the SDK that can pull your templates directly as LangChain's ChatPromptTemplate and PromptTemplate classes.

This makes it much easier to work with chains and LangChain in general.

Example:

template = lunary.get_langchain_template("my-template")

messages = lc_template.format_messages(question="What is the capital of France?")

Make sure to update your SDK to the latest version to use these new methods.

Claude 3 in the playground

Come test your prompts with the new state-of-the-art model. We've found it's better and more concise than GPT-4 in many cases.

Mistral and Claude 3 prices

We've added tracking of Mistral and Claude 3 prices in the dashboard.

Self-hosting

We've removed the dependency on PgGroonga to make it easier to self-host the platform with hosted Postgres services like AWS RDS.

This means Lunary is now also compatible with Postgres 16.

3 projects for free users

We've increased the number of projects that free users can have from 1 to 3. This should help you better organize your work and keep your projects separate.

Parsing of LangChain chains

In the Python SDK, we've pushed numerous improvements to the parsing of LangChain traces. The constant updates to the LangChain format have made it difficult to keep up with the changes, but we're made significant progress in this area.

Fixes

We've fixed a number of issues across the platform where:

• Evaluations would not start correctly from the dashboard
• Editing datasets would stutter and not work correctly
• Deleting a template would not work correctly
• Switching between text and chat templates would break the playground
• Invalid tools would prevent the playground from running
• In JS SDK await lunary.flush not working
• JS SDK crashing in old Node environments because of crypto module