vanillartb/vanilla-prod

By vanillartb

•Updated almost 8 years ago

vanilla-rtb prod box without sources

Image

1.4K

Overview Tags

vanillartb/vanilla-prod repository overview

⁠vanilla-rtb

Real Time Bidding (RTB) - Demand Side Platform framework

open-source library utilizing modern C++11/14/17 features and latest Boost.

What makes us different from other open-source RTB projects on GitHub :

Our stack is fairly small and easy to integrate with any cmake project.
Partitioned targeting data
Monolithic bidder
Code generation for targeting matchers and bidder executable
Minimum dependency on outside vendors.
Decoupled by C++ templates
Very high throughput up to 105K QPS on 16 cores Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
GUI for editing campaign budget

As a model cmake project please visit https://github.com/vanilla-rtb/rapid-bidder⁠

We provide including bindings to popular languages NodeJS/Go/Java/PHP/Python
and custom targetings and bidder executable generators https://github.com/vanilla-rtb/extensions⁠

vanilla-rtb ecosystem⁠

Multi-bidder-model-with-communicator-for-Win-notifications⁠

best performance compared to other stacks - 105K QPS⁠

runs on cloud with docker - see instructions⁠

Recommended build environment: Linux or macOS, CMake >= 3.9.2, GCC >= 7.0, Clang >= 5.0 , Boost >= 1.67

Structure :

/⁠ -- the root directory
- benchmarks/⁠ -- optionaly built benchmarks for IPC caches, json parsers and low overhead IPC audit logger
- CRUD/⁠ -- C++11 high performance HTTP-restful handlers based on boost.ASIO and CRUD API
- docker/⁠ -- vanilla docker files and instructions on how to build and run
- jsonv/⁠ -- DSL mapper of json encoded objects to C++ structures
- parsers/⁠ -- fast zero copy, zero memory allocation parsers
- rapidjson/⁠ -- very fast json tokenizer including SAX / DOM API but requires more memory
- rtb/⁠ -- RTB framework
  - core/⁠ -- generic structures shared in the project ( RTB specific )
  - common/⁠ -- generic RTB agnostic structures
  - datacache/⁠ -- IPC cache generic classes for fast targeting and other lookups
  - exchange/⁠ -- exchange handlers implementation
  - DSL/⁠ -- DSL formats for jsonv , boost::any and rapidjson
- examples/⁠ -- root to our sandbox with examples
  - bidder/⁠ -- collection of application specific classes to support targeting
  - bidder_experimental/⁠ -- bidder based on chained matchers model
  - loader/⁠ -- collection of application specific classes to support campaign loading
  - campaign/⁠ -- add/modify/delete campaign API + UI ( work in progress )
  - datacache/⁠ -- IPC cache implementation based on rtb/datacache model
  - UI/⁠ -- HTML and javascript for campaign budget management
[CMakeLists.txt] - cmake file

The stack of vanilla-rtb includes other C++11/14 projects and is referencing them via gh-subree. To update to the latest version of json-voorhees or CRUD we use the following commands :

git subtree pull --prefix jsonv [email protected]⁠:tgockel/json-voorhees.git master --squash
git subtree pull --prefix CRUD [email protected]⁠:venediktov/CRUD.git master --squash

⁠Creating bidder in 49 lines of code

using Selector = vanilla::ad_selector<vanilla::BudgetManager, Ad>;
using DSLT = DSL::GenericDSL<std::string, DSL::rapid_mapper> ;

//Return from each lambda becomes input for next lambda in the tuple of functions
auto retrieve_domain_f = [&cacheLoader](const std::string& dom, auto&& ...) {
    Domain domain;
    if(!cacheLoader.retrieve(domain,dom)) {
        return boost::optional<uint32_t>();
    }
    return boost::optional<uint32_t>(domain.dom_id);
};

auto retrieve_ico_campaign_f = [&cacheLoader](boost::optional<uint32_t> dom_id, auto&& ...)  {
    std::vector<ICOCampaign> ico_campains;
    if (!cacheLoader.retrieve(ico_campains,*dom_id)) {
        return boost::optional<decltype(ico_campains)>();
    }
    return boost::optional<decltype(ico_campains)>(ico_campains);
};

vanilla::core::Banker<BudgetManager> banker;
auto retrieve_campaign_ads_f = [&](boost::optional<std::vector<ICOCampaign>> campaigns, auto && req, auto && imp)  {
    std::vector<Ad> retrieved_cached_ads;
    for (auto &campaign : *campaigns) {
        if (!cacheLoader.retrieve(retrieved_cached_ads, campaign.campaign_id, imp.banner.get().w, imp.banner.get().h)) {
            continue;
        }
        auto budget_bid = banker.authorize(cacheLoader.get<CampaignCache<BidderConfig>>(), campaign.campaign_id);
        std::transform(std::begin(retrieved_cached_ads),
                       std::end(retrieved_cached_ads),
                       std::begin(retrieved_cached_ads), [budget_bid](Ad & ad){
                          ad.auth_bid_micros = std::min(budget_bid, ad.max_bid_micros);
                          return ad;
                       });
    }
    return retrieved_cached_ads;
};

//creating bidder endpoint utilizing self-referencing pattern
exchange_handler<DSLT> bid_handler(std::chrono::milliseconds(10));
bid_handler
.logger([](const std::string &data) {
    LOG(debug) << "bid request=" << data ;
})
.error_logger([](const std::string &data) {
    LOG(debug) << "bid request error " << data ;
})
.auction_async([&](const BidRequest &request) {
    thread_local vanilla::Bidder<DSLT, Selector> bidder(std::move(Selector()));
    return bidder.bid(request,
                      request.site.get().ref,
                      //chained matchers lambdas defined above
                      retrieve_domain_f,
                      retrieve_ico_campaign_f,
                      retrieve_campaign_ads_f
   );
});

⁠(📗) To build vanilla-rtb use following commands in the root of vanilla-rtb

(installing dependencies before building vanilla stack)⁠

⁠Linux :

$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release .. -G "Unix Makefiles"
$ make -j4 install
####### Creating  Debug build #######
$ cd ..
$ mkdir Debug
$ cd Debug
$ cmake -DCMAKE_BUILD_TYPE=Debug .. -G "Unix Makefiles"
$ make -j4 install

⁠Windows :

same steps as above for linux , only difference is depending on your environment either Visual Studio or NMake project can be used

######### for NMake ####################
cd Release
cmake -DCMAKE_BUILD_TYPE=Release .. -G "NMake Makefiles"
cd ../Debug
cmake -DCMAKE_BUILD_TYPE=Debug   .. -G "NMake Makefiles"
######### for Visual Studio ############
cd Release
cmake -DCMAKE_BUILD_TYPE=Release .. -G "Visual Studio 14 2015"
cd ../Debug
cmake -DCMAKE_BUILD_TYPE=Debug   .. -G "Visual Studio 14 2015"

⁠Mac OS X (Xcode) :

For the reliable results it is suggested to have the build directory out of source tree. The process involves creating a build directory, generating an Xcode project in that directory with CMake, opening the project file generated in the build directory with Xcode, and lastly, adjusting project settings as requried and kicking off the build.

To generate an Xcode project invoke cmake from an empty build directory with command line similar to cmake -G Xcode -DCMAKE_BUILD_TYPE=Release.

⁠Mac OS X (command line tools) :

$ xcode-select --install
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew doctor
$ brew install cmake
$ brew install boost
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release .. -G "Unix Makefiles"
$ make -j4 install

⁠Mac OS X ( with llvm ) :

$ brew doctor
$ brew install cmake
$ brew install boost
$ brew install --with-clang llvm
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/opt/llvm/bin/clang++ -DCMAKE_RANLIB=/usr/local/opt/llvm/bin/llvm-ranlib -DCMAKE_AR=/usr/local/opt/llvm/bin/llvm-ar .. -G "Unix Makefiles"

⁠Parallel Builds

When building on Linux and Mac OS X with Make it's possible to automatically adjust the concurreny of the build using nproc for Linux and sysctl -n hw.physicalcpu for Mac OS X command line tools that returns number of CPUs available to the Make execution context:

Linux :

$ make -j $(nproc) ...

Mac OS X :

make -j $(sysctl -n hw.physicalcpu) ...

It's also possible to specifying the target Load Average with -l flag to prevent machine overloading:

$ make -l $(nproc) ...

And lastly, on Linux, it's possible to run the build with the BATCH scheduling mode (throughput oriented) as:

$ chrt --batch 0 make ...

All above considered the ultimate Make invocation combo on Linux would be something like:

$ chrt --batch 0 make -j$(nproc) -l$(nproc)

⁠Running examples:

⁠Testing simple case - single bidder bound to ip/port

HTTP-Bidder
- vanilla-rtb/Release/examples/bin$ ./http_bidder_test --config etc/config.cfg
test with curl and apache benchmark
- vanilla-rtb/Release/examples/bin$ ./curl.sh --bidder
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --bidder

⁠Testing different json mappers to openrtb::BidRequest available in vanilla-rtb stack

Start Exchange Handler with HTTP handler
- vanilla-rtb/Release/examples/bin$ ./exchange_handler_test --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction-any
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction-rapid

⁠Testing mock-bidders e.g. no bid multi bidders via communicator pattern ( work in progress )

Start Exchange Handler distributing to multi-bidders via communicator
- vanilla-rtb/Release/examples/bin$ ./exchange_handler_test --config etc/config.cfg
Mock-bidders starting multiple in one swoop, currently configured as 5 bidders in config
- vanilla-rtb/Release/examples/bin$ ./mock_bidder_test --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --mock-bidders

⁠Testing multi bidders via communicator pattern ( work in progress )

multi-bidders starting multiple in one swoop (currently configured as 3 bidders in config ) and starting exchange handler
- vanilla-rtb/Release/examples/bin$ ./multi_bidder --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./multi_exchange_handler --config etc/config.cfg

⁠Testing cache loader of ad campaigns and ad budgets ( needs to be extended to get cache update events from outside )

Cache loader
- vanilla-rtb/Release/examples/bin$ ./cache_loader_test --config etc/config.cfg

⁠Testing Mock exchange writen in python

Mock exchange - emulating bid requests
- To run mock exchange you need any python, and python "requests" library installed.
- for simple exchange please run
- vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py
- for various geo and size rotation please run
- vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py --geo "Russia:Moscow USA:NY USA:Washington USA:Chicago" --size '100:300 240:400 420:280'
- for more info please run
- vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py --help

⁠Testing campaign manager paired with Win-notification service

Campaign manager - Budget
- vanilla-rtb/Release/examples/bin$ ./campaign_manager_test --config etc/config.cfg
Notification service and Slave-Banker
- vanilla-rtb/Release/examples/bin$ ./notification_service_test --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./slavebanker_service_test --config etc/config.cfg
To add/delete/modify campign budgets fire up UI by connecting to manager via browser http:://localhost:11081/campaign/index.html