Preamble's Human-Centric Approach to Aligning AI

The eruption in AI capabilities currently taking center stage in discussions from the dinner table to the conference table has everybody looking to get a piece of the action and leverage these new capabilities in their lives and businesses. But what's stopping us from utilizing new AI capabilities? News stories of AI gone awry have already shocked and horrified people, sowing doubt in people's minds and even going as far as fomenting existential dread; not only that, the more capable these tools become, the more opportunities there are for failure.

As with any new technology, where there's an opportunity, there's a risk. But what kind of risks are you taking on? Compliance issues, Liability issues, Performance issues? What kind of AI-related incidents could occur? Do you want to be part of it?

It's important to note that research thus far does not cover the gamut of what is essential to the businesses that may use AI in the future or even individuals. Organizations like Anthropic, OpenAI, Google, Microsoft, Meta, and Amazon may have policies about what is acceptable or not. Still, in the end, we are beholden to their consensus-based approach to acceptable or unacceptable behavior. 

These considerations are only a tiny part of the foundation for what makes these tools useful. Beyond this, developing approaches to deal with bias, toxicity, or unwanted behavior in AI is undoubtedly essential. Still, it misses a very central part of the problem. People don't just want an AI that emulates a decent person; they want an AI that emulates their business-specific values and objectives.

Although these are valuable goals in general, developing and testing these systems typically requires a dedicated team of engineers and labelers.

Through the use of cooperative inverse reinforcement learning (CIRL), natural language interfaces for steering AI, reinforcement learning from human feedback (RLHF), and reinforcement learning with constitutional AI (RL-CAI),  AI can be made much more accessible without the need for any technical know-how. It's only a matter of translating your values-based preferences into policies that steer, manage, and control these systems.

Combining these approaches mitigates many of the drawbacks of any of them on its own. RLHF is notoriously data-hungry, whereas fully automated approaches lack any kind of necessary human oversight and are prone to behavior similar to mode collapse. 

Developing an AI policy marketplace allows for the democratization of these steering values. State-of-the-art technical methods, in concert with an accessible policy marketplace, allows everyone to craft AI that addresses what matters to them. The combination of technology and a transparent marketplace that doesn't require technical skills can enable people to realize their own AI vision without needing to depend on centralized agencies. Without this, AI will remain an opaque and siloed piece of technology, just like content moderation or recommendation systems have been historically. 

There has always been a conflict between appealing to the majority or protecting/serving niche, vulnerable, or specialized populations and use cases. These tools can help to democratize the process of aligning AI for everyone.

Preamble's platform addresses all of these gaps, providing tools for defining what matters to you regarding AI. Not only can you create the type of AI that fits you, but you can also easily use it in a way that suits you. You can have as granular or simple a level of control as you would like, from simply co-opting other entities' policies from the preamble policy marketplace to carefully crafting your own. You can find a way to suit any level of detail with respect to what matters to you. It's easy to monitor whether or not you should be concerned about these policies going stale and integrating them into your current workflows.