โŒ

Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

Problems with gamma function [closed]

I'm not sure if the probabilistic function "gamma" is working correctly.

I have made a model composed only by one service block with a gamma distribution calculated from a real sample and collected the stadistics from this service. When I compare the results from the simulation with the real data they don't fit, their means have a 90% difference. I have calculated the distribution function in Arena Input Analyzer and I have tried using the all the combinations of alpha and beta or 1/beta but none of them give me a result similar to my sample.

I adjunt some screenshoots to show what I am actually dind and collecting.

Distribution obtained in ARENA from real data. Mean and deviation calculated. Model in AnyLogic to test the distribution function Results obtained in AnyLogic

Performing an op:join-inner with op:from-lexicons not working as expected with Optic API

I have the following optic query using the op:from-lexicons access plan:

op:from-lexicons((
        map:map()
            => map:with("lexiconUuid", cts:element-reference(xs:QName("lexiconUuid")))
            => map:with("elementInDocumentToFilter", cts:element-reference(xs:QName("elementInDocumentToFilter")))
            => map:with("documentUuid", cts:element-reference(xs:QName("documentUuid")))
    ),"lexiconView")
    => op:where(op:eq(op:view-col("lexiconView","elementInDocumentToFilter"), "Filter Value"))
    => op:group-by(op:view-col("lexiconView","lexiconUuid"), 
        op:count("countOfDocsMeetingFilter", "documentUuid", map:entry('values', 'distinct'))
    )
    => op:select((
        op:view-col("lexiconView","lexiconUuid"),
        op:as("hasCountOver0", op:gt(op:col("countOfDocsMeetingFilter"), 0))
    ))

When I add op:result() to this query I get an output like:

lexiconView.lexiconUuid hasCountOver0
lexicon-uuid-1 true
lexicon-uuid-2 false

This query is using the existing indexes to return the lexiconUuid and whether or not it appears in any documents based on the filter. When I try to use the query above as the $rightPlan for an op:join-inner I get no results (as in, no join took place). Here is my query using the lexicon query above:

op:from-view("schema", "table1")
    (: table2 contains the lexiconUuid element I want to join the lexicon query with :)
    => op:join-inner(op:from-view("schema", "table2"),
        op:on(
            op:view-col("table1", "someUuid"),
            op:view-col("table2", "someUuid")
        )
    )
    => op:join-inner(
        (: op:from-lexicons() access plan here :),
        op:on(
            op:view-col("table2", "lexiconUuid"),
            op:view-col("lexiconView", "lexiconUuid")
        )
    )

I know for a fact that the lexiconUuid is present after joining table2 and exists in the op:from-lexicons query (i.e. lexicon-uuid-1 exists in table2 so it should join on the "lexiconView" without a problem). I replicated this "lexiconView" as a TDE and the join performs as expected but requires an aggregation on the existing data so I have to op:group-by() all of the columns in table1 and table2. I was hoping to be able to use the lexicon to avoid having another TDE and performing a group-by/aggregation on more columns. Is there something I am missing about using the op:from-lexicons access plan? Can it be used in joins? Has anyone ran into this before?

MarkLogic Optic API, existsJoin with data selection using empty qualifier resulting in 'Ambiguous column reference' error

I came across following situation - I use Optic API, I have first view filtered by second one using existsJoin to get only rowset I want. Then I need to include property which was used in op.on definition (the same column exists in both views) in objects which I return as a result, additionally that property must be present without any qualifiers etc.

Below is the simplest possible code that mimics what I need to do in mine application (using views created by TDE).

const op = require('/MarkLogic/optic');
const testViewToFilter = op.fromLiterals([
  {id:1000, val: 10},
  {id:1001, val: 11},
  {id:1002, val: 12}
], 'testViewToFilter');
const filteringView = op.fromLiterals([{id:1000}], 'filteringView')
testViewToFilter
  .existsJoin(
    filteringView,
    op.on(
      testViewToFilter.col('id'),
      filteringView.col('id')
    )
  )
  .select([testViewToFilter.col('id')], '')
  .result()

What I expect:

[{id: 1000}]

What I get:

[javascript] SQL-AMBCOLUMN: return plan.execute(query); -- Ambiguous column reference: found filteringView.id and id

Am I doing something wrong? How can I achieve the result I expect? I don't understand why do I get Ambiguous column reference for id column.

I get the same results for MarkLogic versions: 10.0-9.2 and 10.0-9.5

Optic API vs CTS query performance

I am using below two queries(Optic and CTS) to get the values for path-range index /tXML/Item/PutawayCategory.

Query 1: - It took approx. 4 milliseconds to execute and return 17 distinct values. I tried executing this same query multiple times.

xquery version "1.0-ml";
import module namespace op="http://marklogic.com/optic"  at "/MarkLogic/optic.xqy";

op:from-lexicons(map:entry("PutawayCategory", cts:path-reference("/tXML/Item/PutawayCategory")))
=> op:where-distinct()
=> op:result()

Query 2: - It took approx. 0.30 milliseconds to get the same result as Query 1

xquery version "1.0-ml";

cts:values(cts:path-reference("/tXML/Item/PutawayCategory"))

I am not getting why Optic Query is taking more time to execute than cts query.

Please help me to understand this.

Minus function in Optic Query

I have requirement to do minus in one of the use case but in MarkLogic I am not able to use minus function. Is there any alternate way to do this?

select table1.value1 from table1 where table1.date = '2020-11-27'
minus
select table1.value1 from table1 where table1.date = '2020-11-26'

MarkLogic optic query using two indexes returns no results

I want to use the MarkLogic optic API to join two range indexes but somehow they don't join. Is the query I wrote wrong or can't I compare the indexes used?

I have two indexes defined:

  • an element-attribute range index x/@refid
  • a range field index 'id'

Both are of type string and have the same collation defined. Both indexes have data that I can retrieve with cts:values() function. Both are huge indexes and I want to join them using optics so I have constructed the following query :

import module namespace op="http://marklogic.com/optic"
at "/MarkLogic/optic.xqy";

let $subfrag := op:fragment-id-col("subfrag") 
let $notfrag := op:fragment-id-col("notfrag") 

let $query :=
cts:and-query((
  cts:collection-query("latest")
))

let $subids := op:from-lexicons(
   map:entry("subid", cts:field-reference("id")), (), $subfrag) => op:where($query)

let $notids := op:from-lexicons(
   map:entry("notid", cts:element-attribute-reference(xs:QName("x"), xs:QName("refid"))),
   (),
       $notfrag)
    
return $subids
   => op:join-cross-product($notids)
   => op:where(op:eq($notfrag, $subfrag))
   => op:result()

This query uses the join-cross-product and when I remove the op:where clause I get all values left and right. I verified and some are equal so the clause should filter only those rows i'm actually interested in. But somehow it doesn't work and I get an empty result. Also, if I replace one of the values in the op:eq with a string value it doesn't return a result.

When I use the same variable in the op:eq operator (like op:eq($notfrag, $notfrag)) I get results back so the statement as is works. Just not the comparison between the two indexes.

I have also used variants with join-inner and left-outer-join but those are also returning no results.

Am I comparing two incomparable indexes or am I missing some statement (as documentation/example is a bit thin).

(of course I can solve by not using optics but in this case it would be a perfect fit)

[update]

I got it working by eventually by changing the final statement:

return $subids
=> op:join-cross-product($notids)
=> op:where(op:eq(op:col('subid'), op:col('notid')))
=> op:result()

So somehow you cannot use the fragment definitions in the condition. After this I replaced the join-cross-product with a join-inner construction which should be a bit more efficient.

And to be complete, I initially used the example from the MarkLogic documentation found here (https://docs.marklogic.com/guide/app-dev/OpticAPI#id_87356), specifically the last example where they use a fragment column definition to be used as param in the join-inner statement that didn't work in my case.

Optic API: how do I get the value of a column that goes with a max value?

My data set has two arrays holding objects with two properties, a date and a value. For each array, I need to get the object that has the most recent date. I'm trying to do this from indexes and am exploring using the Optic API for the query.

My view has three columns: "statusType", which indicates which array the values come from; "statusDate"; and "value". With the following query, I can get the most recent date for each type, but I don't see how I can get the value that goes with it.

const op = require('/MarkLogic/optic');

op.fromView('Parent', 'info')
  .where(cts.documentQuery('/test/doc1.json'))
  .groupBy([op.col('statusType')], [op.max('maxdate', op.col('statusDate'))])
  .result()

Produces:

{
  "statusType": "subtype1", 
  "maxdate": "2020-09-29T16:33:18.6301434-04:00"
},
{
  "statusType": "subtype2", 
  "maxdate": "2020-08-29T16:33:18.6301434-04:00"
}

If I add value to the first parameter to groupBy, I get all distinct combinations of type and value (with the maxdate). If I add value to the second parameter to groupBy, I get the last value, not the one associated with the maxdate.

Expected output:

{
  "statusType": "subtype1", 
  "value": "valueB",
  "maxdate": "2020-09-29T16:33:18.6301434-04:00"
},
{
  "statusType": "subtype2", 
  "value": "valueC",
  "maxdate": "2020-08-29T16:33:18.6301434-04:00"
}

Sample data:

'use strict';
declareUpdate();

xdmp.documentInsert(
  '/test/doc1.json',
  {
    "parent": {
      "subtype1": [
        {
          "value": "valueA", 
          "date": "2020-07-29T16:33:18.6301434-04:00"
        }, 
        {
          "value": "valueB", 
          "date": "2020-09-29T16:33:18.6301434-04:00"
        }
      ], 
      "subtype2": [
        {
          "value": "valueC", 
          "date": "2020-08-29T16:33:18.6301434-04:00"
        }, 
        {
          "value": "valueD", 
          "date": "2020-07-29T16:33:18.6301434-04:00"
        }
      ]
    }
  }
)

Template 1:

declareUpdate();
const tde = require("/MarkLogic/tde.xqy");

let template = 
  xdmp.toJSON(
    {
      "template": {
        "context": "/parent/subtype1",
        "rows": [
          {
            "schemaName": "Parent",
            "viewName": "info",
            "columns": [
              {
                "name": "statusType",
                "scalarType": "string",
                "val": "'subtype1'"
              },
              {
                "name": "value",
                "scalarType": "string",
                "val": "value"
              },
              {
                "name": "statusDate",
                "scalarType": "dateTime",
                "val": "date"
              }
            ]
          }
        ]
      }
    }
  );

// comment and uncomment based on which action you want to take
let action = 
  //'validate'
  //'extract'
  'insert'
;

if (action === 'validate') {
  tde.validate([template]);
} else if (action === 'extract') {
  tde.nodeDataExtract([cts.doc( "/test/doc1.json" )],  [template])
} else if (action === 'insert') {
  tde.templateInsert("/tde/subtype1.json", template, xdmp.defaultPermissions(), ["TDE"])
}

Template 2:

declareUpdate();
const tde = require("/MarkLogic/tde.xqy");

let template = 
  xdmp.toJSON(
    {
      "template": {
        "context": "/parent/subtype2",
        "rows": [
          {
            "schemaName": "Parent",
            "viewName": "info",
            "columns": [
              {
                "name": "statusType",
                "scalarType": "string",
                "val": "'subtype2'"
              },
              {
                "name": "value",
                "scalarType": "string",
                "val": "value"
              },
              {
                "name": "statusDate",
                "scalarType": "dateTime",
                "val": "date"
              }
            ]
          }
        ]
      }
    }
  );

// comment and uncomment based on which action you want to take
let action = 
  //'validate'
  //'extract'
  'insert'
;

if (action === 'validate') {
  tde.validate([template]);
} else if (action === 'extract') {
  tde.nodeDataExtract([cts.doc( "/test/doc1.json" )],  [template])
} else if (action === 'insert') {
  tde.templateInsert("/tde/subtype2.json", template, xdmp.defaultPermissions(), ["TDE"])
}

How do I use a parameter value, set by a user in a web application, in an Optic API query in Marklogic?

Blockquote

I have a web application (written in js) where a user can enter a name (along with other criteria) and a search against a ML database is performed and a tabular representation of some of the data associated to that name (or other search criteria) is returned. In the first stages of development I wrote optic api queries with the value for name hard coded in the where statement of the query. I did this so I could test that that the REST api was calling the query properly, etc. That all is working and I can see the right results for that hard coded name being returned in the web app. Now I want to pass the user supplied value for name to the Optic api query (EntityInformation_Optic_API_Query.json) to replace the hard coded value in where statement. How do I do this? Thanks in advance.

owner edit

The application is written in PHP and I have copied code below. In this you can see where the optic api query is called and where the parameters are built.

<?php
if (!defined('RSS_BASE_URL')) exit;

use MarkLogic\MLPHP as MLPHP;

function rss_api_call($json, $params = array(), $resource = 'rows', $verb = 'POST')
{
    $json = __DIR__ . '/json/' . $json;

    if (!file_exists($json)) return false;

    $request = new MLPHP\RESTRequest('POST', 'rows', $params, file_get_contents($json), array('Content-type' => 'application/json'));
    $response = rss_get_client()->send($request);

    return json_decode($response->getBody());
}

function rss_api_entities()
{
    $params = array();

    return rss_api_call('EntityInformation_Optic_API_Query.json', $params);
    //return (empty($params)) ? '' : rss_api_call('EntityInformation_Optic_API_Query.json', $params);
}

function rss_api_search()
{
    $params = array();

    if (isset($_POST['companyname']) && !empty($_POST['companyname']))
    {
        $params['CompanyName'] = htmlspecialchars($_POST['companyname']);
    }

    if (isset($_POST['EIN']) && !empty($_POST['ein']))
    {
        $params['EIN'] = htmlspecialchars($_POST['ein']);
    }

    if (isset($_POST['city']) && !empty($_POST['city']))
    {
        $params['EntityCity'] = htmlspecialchars($_POST['city']);
    }

    if (isset($_POST['state']) && !empty($_POST['state']))
    {
        $params['EntityState'] = htmlspecialchars($_POST['state']);
    }

    if (isset($_POST['zip']) && !empty($_POST['zip']))
    {
        $params['EntityZip'] = htmlspecialchars($_POST['zip']);
    }

    return rss_api_call('SearchResults_Optic_API_Query.json', $params);
    //return (empty($params)) ? '' : rss_api_call('SearchResults_Optic_API_Query.json', $params);
}

function rss_asset_path($path, $file_name)
{
    global $rss_manifest;

    if (empty($rss_manifest))
    {
        ob_start();

        include(__DIR__ . '/../manifest.json');

        $rss_manifest = json_decode(ob_get_clean(), true);
    }

    if (isset($rss_manifest[$file_name]))
    {
        $file_name = $rss_manifest[$file_name];
    }

    return RSS_ASSET_PATH . $path . '/' . $file_name;
}

function rss_box($title, $content)
{
    echo '<div class="rss-box">' .
        '<div class="rss-box-title">' . $title . '</div>' .
        '<div class="rss-box-content">' . $content . '</div>' .
        '</div>';
}

function rss_get_client()
{
    $mlphp = new MLPHP\MLPHP(array
    (
        'host' => RSS_API_HOST,
        'port' => RSS_API_PORT,
        'version' => RSS_API_VERSION,
        'username' => RSS_API_USERNAME,
        'password' => RSS_API_PASSWORD
    ));

    return $mlphp->getClient();
}

function rss_hidden_search_fields()
{
    echo '<div class="rss-hidden">';

    $fields = array('debug', 'companyname', 'ein', 'city', 'state', 'zip');

    foreach ($fields as $field)
    {
        echo (isset($_POST[$field])) ? '<input name="' . $field . '" type="hidden" value="' . htmlspecialchars($_POST[$field]) . '" />' : '';
    }

    if (!empty($_POST['social']))
    {
        $social = (is_array($_POST['social'])) ? $_POST['social'] : array($_POST['social']);

        foreach ($social as $social_network)
        {
            echo '<input name="social[]" type="hidden" value="' . htmlspecialchars($social_network) . '" />';
        }
    }

    echo (empty($_POST['social-select-all'])) ? '' : '<input name="social-select-all" type="hidden" value="1" />';
    echo '</div>';
}



Part of the EntityInformation_Optic_API_Query.json is below
-----

edit (information from a comment below):

The request will be a POST to /v1/rows. Does the following payload look correct? 

```javascript
{
    "$optic": {
        "ns": "op",
        "fn": "operators",
        "args": [
            {
                "ns": "op",
                "fn": "from-view",
                "args": [ "TestSchema", "SUT", null, null ]
            },
            {
                "ns": "op",
                "fn": "where",
                "args": [
                    {
                        "ns": "op",
                        "fn": "eq",
                        "args": [
                            {
                                "ns": "op",
                                "fn": "col",
                                "args": [ "CompanyName" ]
                            },
                            "${req.params.CompanyName}"
                        ]
                    }
                ]
            }
โŒ
โŒ