美文网首页
Oozie-参数表达式体系

Oozie-参数表达式体系

作者: PunyGod | 来源:发表于2016-07-04 17:11 被阅读1405次

ELService 服务是用于初始化el计算的服务,oozie的参数计算使用的是 org.apache.commons.el 的 JSP Expression Language Evaluator。功能可以做到,变量值的动态计算,支持oozie这次的计算表达式。这些表达式是固定的,需要在初始化的时候加载到oozie中。现在来看看代码:

oozie将对参数的计算封装成 ELEvaluator

private Context context;
private ExpressionEvaluatorImpl evaluator = new ExpressionEvaluatorImpl();

ExpressionEvaluatorImpl ** 是common包中的计算实现。context** 继承了
VariableResolver,FunctionMapper

/**
 * Provides functions and variables for the EL evaluator. <p/> All functions and variables in the context of an EL
 * evaluator are accessible from EL expressions.
 */
public static class Context implements VariableResolver, FunctionMapper {
    private Map<String, Object> vars;
    private Map<String, Method> functions;
    /**
     * Create an empty context.
     */
    public Context() {
        vars = new HashMap<String, Object>();
        functions = new HashMap<String, Method>();
    }
    /**
     * Add variables to the context. <p/>
     *
     * @param vars variables to add to the context.
     */
    public void setVariables(Map<String, Object> vars) {
        this.vars.putAll(vars);
    }
    /**
     * Add a variable to the context. <p/>
     *
     * @param name variable name.
     * @param value variable value.
     */
    public void setVariable(String name, Object value) {
        vars.put(name, value);
    }
    /**
     * Return a variable from the context. <p/>
     *
     * @param name variable name.
     * @return the variable value.
     */
    public Object getVariable(String name) {
        return vars.get(name);
    }
    /**
     * Add a function to the context. <p/>
     *
     * @param prefix function prefix.
     * @param functionName function name.
     * @param method method that will be invoked for the function, it must be a static and public method.
     */
    public void addFunction(String prefix, String functionName, Method method) {
        if ((method.getModifiers() & (Modifier.PUBLIC | Modifier.STATIC)) != (Modifier.PUBLIC | Modifier.STATIC)) {
            throw new IllegalArgumentException(XLog.format("Method[{0}] must be public and static", method));
        }
        prefix = (prefix.length() > 0) ? prefix + ":" : "";
        functions.put(prefix + functionName, method);
    }
    /**
     * Resolve a variable name. Used by the EL evaluator implemenation. <p/>
     *
     * @param name variable name.
     * @return the variable value.
     * @throws ELException thrown if the variable is not defined in the context.
     */
    public Object resolveVariable(String name) throws ELException {
        if (!vars.containsKey(name)) {
            throw new ELException(XLog.format("variable [{0}] cannot be resolved", name));
        }
        return vars.get(name);
    }
    /**
     * Resolve a function prefix:name. Used by the EL evaluator implementation. <p/>
     *
     * @param prefix function prefix.
     * @param name function name.
     * @return the method associated to the function.
     */
    public Method resolveFunction(String prefix, String name) {
        if (prefix.length() > 0) {
            name = prefix + ":" + name;
        }
        return functions.get(name);
    }
}

这里的 Map中的key是方法或者是数值的别名。

/**
 * Evaluate an EL expression. <p/>
 *
 * @param expr EL expression to evaluate.
 * @param clazz return type of the EL expression.
 * @return the object the EL expression evaluated to.
 * @throws Exception thrown if an EL function failed due to a transient error or EL expression could not be
 * evaluated.
 */
@SuppressWarnings({"unchecked", "deprecation"})
public <T> T evaluate(String expr, Class<T> clazz) throws Exception {
    ELEvaluator existing = current.get();
    try {
        current.set(this);
        return (T) evaluator.evaluate(expr, clazz, context, context);
    }
    catch (ELException ex) {
        if (ex.getRootCause() instanceof Exception) {
            throw (Exception) ex.getRootCause();
        }
        else {
            throw ex;
        }
    }
    finally {
        current.set(existing);
    }
}

表达式计算的具体实现方法;

正如前面介绍组件的时候,可以看到oozie内置了大量的表达式运算。现在我们来看看他们的具体实现;

<!-- Workflow specifics -->
    <property>
        <name>oozie.service.ELService.constants.workflow</name>
        <value>
            KB=org.apache.oozie.util.ELConstantsFunctions#KB,
            MB=org.apache.oozie.util.ELConstantsFunctions#MB,
            GB=org.apache.oozie.util.ELConstantsFunctions#GB,
            TB=org.apache.oozie.util.ELConstantsFunctions#TB,
            PB=org.apache.oozie.util.ELConstantsFunctions#PB,
            RECORDS=org.apache.oozie.action.hadoop.HadoopELFunctions#RECORDS,
            MAP_IN=org.apache.oozie.action.hadoop.HadoopELFunctions#MAP_IN,
            MAP_OUT=org.apache.oozie.action.hadoop.HadoopELFunctions#MAP_OUT,
            REDUCE_IN=org.apache.oozie.action.hadoop.HadoopELFunctions#REDUCE_IN,
            REDUCE_OUT=org.apache.oozie.action.hadoop.HadoopELFunctions#REDUCE_OUT,
            GROUPS=org.apache.oozie.action.hadoop.HadoopELFunctions#GROUPS 
       </value>
        <description>
            EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT.
        </description>
    </property>
<property>
    <name>oozie.service.ELService.functions.workflow</name>
    <value>
        firstNotNull=org.apache.oozie.util.ELConstantsFunctions#firstNotNull,
        concat=org.apache.oozie.util.ELConstantsFunctions#concat,
        replaceAll=org.apache.oozie.util.ELConstantsFunctions#replaceAll,
        appendAll=org.apache.oozie.util.ELConstantsFunctions#appendAll,
        trim=org.apache.oozie.util.ELConstantsFunctions#trim,
        timestamp=org.apache.oozie.util.ELConstantsFunctions#timestamp,
        urlEncode=org.apache.oozie.util.ELConstantsFunctions#urlEncode,        toJsonStr=org.apache.oozie.util.ELConstantsFunctions#toJsonStr,
        toPropertiesStr=org.apache.oozie.util.ELConstantsFunctions#toPropertiesStr,
        toConfigurationStr=org.apache.oozie.util.ELConstantsFunctions#toConfigurationStr,
        wf:id=org.apache.oozie.DagELFunctions#wf_id,
        wf:name=org.apache.oozie.DagELFunctions#wf_name,
        wf:appPath=org.apache.oozie.DagELFunctions#wf_appPath,
        wf:conf=org.apache.oozie.DagELFunctions#wf_conf,
        wf:user=org.apache.oozie.DagELFunctions#wf_user,
        wf:group=org.apache.oozie.DagELFunctions#wf_group,
        wf:callback=org.apache.oozie.DagELFunctions#wf_callback,
        wf:transition=org.apache.oozie.DagELFunctions#wf_transition,
        wf:lastErrorNode=org.apache.oozie.DagELFunctions#wf_lastErrorNode,
        wf:errorCode=org.apache.oozie.DagELFunctions#wf_errorCode,
        wf:errorMessage=org.apache.oozie.DagELFunctions#wf_errorMessage,
        wf:run=org.apache.oozie.DagELFunctions#wf_run,
        wf:actionData=org.apache.oozie.DagELFunctions#wf_actionData,
        wf:actionExternalId=org.apache.oozie.DagELFunctions#wf_actionExternalId,
        wf:actionTrackerUri=org.apache.oozie.DagELFunctions#wf_actionTrackerUri,
        wf:actionExternalStatus=org.apache.oozie.DagELFunctions#wf_actionExternalStatus,
        hadoop:counters=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_counters,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf,
        fs:exists=org.apache.oozie.action.hadoop.FsELFunctions#fs_exists,
        fs:isDir=org.apache.oozie.action.hadoop.FsELFunctions#fs_isDir,
        fs:dirSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_dirSize,
        fs:fileSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_fileSize,
        fs:blockSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_blockSize,
        hcat:exists=org.apache.oozie.coord.HCatELFunctions#hcat_exists
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>

现在看几个具体实现:

wf:user=org.apache.oozie.DagELFunctions#wf_user,

/**
 * Return the job owner user name.
 *
 * @return the job owner user name.
 */
public static String wf_user() {
    return getWorkflow().getUser();
}

public static WorkflowJobBean getWorkflow() {
    ELEvaluator eval = ELEvaluator.getCurrent();
    return (WorkflowJobBean) eval.getVariable(WORKFLOW);
}


  hadoop:counters=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_counters


public static Map<String, Map<String, Long>> hadoop_counters(String nodeName) throws ELEvaluationException {
    WorkflowInstance instance = DagELFunctions.getWorkflow().getWorkflowInstance();
    Object obj = instance.getTransientVar(nodeName + WorkflowInstance.NODE_VAR_SEPARATOR + HADOOP_COUNTERS);
    Map<String, Map<String, Long>> counters = (Map<String, Map<String, Long>>) obj;
    if (counters == null) {
        counters = getCounters(nodeName);
        // In Hadoop 0.23 they deprecated 'org.apache.hadoop.mapred.Task$Counter' and they REMOVED IT
        // Here we are getting the new Name and inserting it using the old name if the old name is not found
        if (counters.get(RECORDS) == null) {
            counters.put(RECORDS, counters.get(RECORDS_023));
        }
        instance.setTransientVar(nodeName + WorkflowInstance.NODE_VAR_SEPARATOR + HADOOP_COUNTERS, counters);
    }
    return counters;
}


fs:exists=org.apache.oozie.action.hadoop.FsELFunctions#fs_exists

/**
 * Return if a path exists.
 *
 * @param pathUri file system path uri.
 * @return <code>true</code> if the path exists, <code>false</code> if it does not.
 * @throws Exception
 */
public static boolean fs_exists(String pathUri) throws Exception {
    URI uri = new URI(pathUri);
    String path = uri.getPath();
    FileSystem fs = getFileSystem(uri);
    return fs.exists(new Path(path));
}



hcat:exists=org.apache.oozie.coord.HCatELFunctions#hcat_exists


/**
 * Return true if partitions exists or false if not.
 *
 * @param uri hcatalog partition uri.
 * @return <code>true</code> if the uri exists, <code>false</code> if it does not.
 * @throws Exception
 */
public static boolean hcat_exists(String uri) throws Exception {
    URI hcatURI = new URI(uri);
    URIHandlerService uriService = Services.get().get(URIHandlerService.class);
    URIHandler handler = uriService.getURIHandler(hcatURI);
    WorkflowJob workflow = DagELFunctions.getWorkflow();
    String user = workflow.getUser();
    return handler.exists(hcatURI, EMPTY_CONF, user);
}

   <!-- Resolve SLA information during Workflow job submission --><property>
       <name>oozie.service.ELService.constants.wf-sla-submit</name>
       <value>
           MINUTES=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_MINUTES,
           HOURS=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_HOURS,
           DAYS=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_DAYS
           </value>
       <description>
           EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT.
       </description>
   </property>


<property>
    <name>oozie.service.ELService.functions.coord-job-submit-freq</name>
    <value>
        coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days,
        coord:months=org.apache.oozie.coord.CoordELFunctions#ph1_coord_months,
        coord:hours=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hours,
        coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes,
        coord:endOfDays=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfDays,
        coord:endOfMonths=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfMonths,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,       
 coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>

现在看几个具体实现:

coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days

public static int ph1_coord_days(int val) {
    val = ParamChecker.checkGTZero(val, "n");
    ELEvaluator eval = ELEvaluator.getCurrent();
    eval.setVariable("timeunit", TimeUnit.DAY);
    eval.setVariable("endOfDuration", TimeUnit.NONE);
    return val;
}

<property>
    <name>oozie.service.ELService.functions.coord-job-wait-timeout</name>
    <value>
        coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days,
        coord:months=org.apache.oozie.coord.CoordELFunctions#ph1_coord_months,
        coord:hours=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hours,
        coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>


<!-- EL Evalautor setup to resolve mainly all constants/variables - no EL functions is resolved -->
    <property>
        <name>oozie.service.ELService.constants.coord-job-submit-nofuncs</name>
        <value>
            MINUTE=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTE,
            HOUR=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOUR,
            DAY=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAY,
            MONTH=org.apache.oozie.coord.CoordELConstants#SUBMIT_MONTH,
            YEAR=org.apache.oozie.coord.CoordELConstants#SUBMIT_YEAR
        </value>
        <description>
            EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT.
        </description>
    </property>


<property>
    <name>oozie.service.ELService.functions.coord-job-submit-nofuncs</name>
    <value>
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
        coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>


<property>
    <name>oozie.service.ELService.functions.coord-job-submit-instances</name>
    <value>
        coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hoursInDay_echo,
        coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph1_coord_daysInMonth_echo,
        coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_tzOffset_echo,
        coord:current=org.apache.oozie.coord.CoordELFunctions#ph1_coord_current_echo,
        coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_currentRange_echo,
        coord:offset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_offset_echo,
        coord:latest=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latest_echo,
        coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latestRange_echo,
        coord:future=org.apache.oozie.coord.CoordELFunctions#ph1_coord_future_echo,
        coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_futureRange_echo,
        coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
        coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,       
 coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph1_coord_absolute_echo,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>

现在看几个具体实现:

coord:current=org.apache.oozie.coord.CoordELFunctions#ph1_coord_current_echo

public static String ph1_coord_current_echo(String n) {
    return echoUnResolved("current", n);
}

private static String echoUnResolved(String functionName, String n) {
    return echoUnResolvedPre(functionName, n, "coord:");
}

private static String echoUnResolvedPre(String functionName, String n, String prefix) {
    ELEvaluator eval = ELEvaluator.getCurrent();
    eval.setVariable(".wrap", "true");
    return prefix + functionName + "(" + n + ")"; // Unresolved
}


<property>
    <name>oozie.service.ELService.functions.coord-job-submit-data</name>
    <value>
        coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataIn_echo,
        coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataOut_echo,
        coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap,
        coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actualTime_echo_wrap,
        coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo,
        coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo,
        coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,
        coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo,
        coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, 
       coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseIn_echo,
        coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseOut_echo,
        coord:tableIn=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableIn_echo,
        coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableOut_echo,
        coord:dataInPartitionFilter=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionFilter_echo,
        coord:dataInPartitionMin=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionMin_echo,
        coord:dataInPartitionMax=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionMax_echo,
        coord:dataInPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitions_echo,
        coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitions_echo,
        coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitionValue_echo,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>

现在看几个具体实现:

coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataIn_echo

public static String ph1_coord_dataIn_echo(String n) {
    ELEvaluator eval = ELEvaluator.getCurrent();
    String val = (String) eval.getVariable("oozie.dataname." + n);
    if (val == null || val.equals("data-in") == false) {
        XLog.getLog(CoordELFunctions.class).error("data_in_name " + n + " is not valid");
        throw new RuntimeException("data_in_name " + n + " is not valid");
    }
    return echoUnResolved("dataIn", "'" + n + "'");
}

<property>
       <name>oozie.service.ELService.constants.coord-sla-submit</name>
       <value>
           MINUTES=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTES,
           HOURS=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOURS,
           DAYS=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAYS
           </value>
       <description>
           EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT.
       </description>
   </property>


<property>
    <name>oozie.service.ELService.functions.coord-sla-submit</name>
    <value>        coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataOut_echo,
        coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed,
        coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actualTime_echo_wrap,
        coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo,
        coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo,
        coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,
        coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo,
        coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
       coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseOut_echo,
        coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableOut_echo,
        coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitions_echo,
        coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitionValue_echo,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>


<property>
    <name>oozie.service.ELService.functions.coord-action-create</name>
    <value>
        coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay,
        coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph2_coord_daysInMonth,
        coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_tzOffset,
        coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current,
        coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_currentRange,
        coord:offset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_offset,
        coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,
        coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latestRange_echo,
        coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo,
        coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_futureRange_echo,
        coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actionId,
        coord:name=org.apache.oozie.coord.CoordELFunctions#ph2_coord_name,
        coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
        coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_echo,
        coord:absoluteRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_range,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>

现在看几个具体实现:

coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay


/**
 * Determine how many hours is on the date of n-th dataset instance. <p/> It depends on: <p/> 1. Data set frequency
 * <p/> 2. Data set Time unit (day, month, minute) <p/> 3. Data set Time zone/DST <p/> 4. End Day/Month flag <p/> 5.
 * Data set initial instance <p/> 6. Action Creation Time
 *
 * @param n instance count <p/> domain: n is integer
 * @return number of hours on that day <p/> returns -1 means n-th instance is earlier than Initial-Instance of DS
 * @throws Exception 
*/

public static int ph2_coord_hoursInDay(int n) throws Exception {
    int datasetFrequency = (int) getDSFrequency();
    // /Calendar nominalInstanceCal =
    // getCurrentInstance(getActionCreationtime());
    Calendar nominalInstanceCal = getEffectiveNominalTime();
    if (nominalInstanceCal == null) {
        return -1;
    }
    nominalInstanceCal.add(getDSTimeUnit().getCalendarUnit(), datasetFrequency * n);
    /*
     * if (nominalInstanceCal.getTime().compareTo(getInitialInstance()) < 0)
     * { return -1; } 
    */
    nominalInstanceCal.setTimeZone(getDatasetTZ());// Use Dataset TZ
    // DateUtils.moveToEnd(nominalInstanceCal, getDSEndOfFlag());
    return DateUtils.hoursInDay(nominalInstanceCal);
}


coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current


/**
 * Determine the date-time in Oozie processing timezone of n-th dataset instance. <p/> It depends on: <p/> 1.
 * Data set frequency <p/> 2.
 * Data set Time unit (day, month, minute) <p/> 3. Data set Time zone/DST <p/> 4. End Day/Month flag <p/> 5. Data
 * set initial instance <p/> 6. Action Creation Time
 *
 * @param n instance count domain: n is integer
 * @return date-time in Oozie processing timezone of the n-th instance returns 'null' means n-th instance is
 * earlier than Initial-Instance of DS
 * @throws Exception
 */
public static String ph2_coord_current(int n) throws Exception {
    if (isSyncDataSet()) {
 // For Sync Dataset
        return coord_current_sync(n);
    }
    else {
        throw new UnsupportedOperationException("Asynchronous Dataset is not supported yet");
    }
}



coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime


/**
 * Convert from standard date-time formatting to a desired format.
 * <p/>
 * @param dateTimeStr - A timestamp in standard (ISO8601) format.
 * @param format - A string representing the desired format.
 * @return coordinator action creation or materialization date time
 * @throws Exception if unable to format the Date object to String */
public static String ph2_coord_formatTime(String dateTimeStr, String format)
        throws Exception {
    Date dateTime = DateUtils.parseDateOozieTZ(dateTimeStr);
    return DateUtils.formatDateCustom(dateTime, format);
}



<property>
    <name>oozie.service.ELService.functions.coord-action-create-inst</name>
    <value>
        coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay,
        coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph2_coord_daysInMonth,
        coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_tzOffset,
        coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current_echo,
        coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_currentRange_echo,
        coord:offset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_offset_echo,
        coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,
        coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latestRange_echo,
        coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo,
        coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_futureRange_echo,
        coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,        
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_echo,
        coord:absoluteRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_range,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>



<property>
    <name>oozie.service.ELService.functions.coord-action-start</name>
    <value>
        coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph3_coord_hoursInDay,
        coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph3_coord_daysInMonth,
        coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_tzOffset,
        coord:latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest,
        coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latestRange,
        coord:future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future,
        coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph3_coord_futureRange,
        coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataIn,
        coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataOut,
        coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime,
        coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actualTime,
        coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset,
        coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateTzOffset,
        coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime,
        coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actionId,
        coord:name=org.apache.oozie.coord.CoordELFunctions#ph3_coord_name,
        coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
       coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user,
        coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseIn,
        coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseOut,
        coord:tableIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableIn,
        coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableOut,
        coord:dataInPartitionFilter=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionFilter,
        coord:dataInPartitionMin=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionMin,
        coord:dataInPartitionMax=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionMax,
        coord:dataInPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitions,
        coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitions,
        coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitionValue,
        hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf
    </value>
    <description>
        EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
    </description>
</property>

几个具体实现

coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseIn

public static String ph3_coord_databaseIn(String dataName) {
    HCatURI hcatURI = getURIFromResolved(dataName, EventType.input);
    if (hcatURI != null) {
        return hcatURI.getDb();
    }
    else {
        return "";
    }
}


coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseOut


public static String ph3_coord_databaseOut(String dataName) {
    HCatURI hcatURI = getURIFromResolved(dataName, EventType.output);
    if (hcatURI != null) {
        return hcatURI.getDb();
    }
    else {
        return "";
    }
}



private static HCatURI getURIFromResolved(String dataInName, EventType type) {
    final XLog LOG = XLog.getLog(HCatELFunctions.class);
    StringBuilder uriTemplate = new StringBuilder();
    ELEvaluator eval = ELEvaluator.getCurrent();
    String uris;
    if(type == EventType.input) {
        uris = (String) eval.getVariable(".datain." + dataInName);
    }
    else { //type=output
        uris = (String) eval.getVariable(".dataout." + dataInName);
    }
    if (uris != null) {
        String[] uri = uris.split(CoordELFunctions.DIR_SEPARATOR, -1);
        uriTemplate.append(uri[0]);
    }
    else {
        LOG.warn("URI is NULL");
        return null;
    }
    LOG.info("uriTemplate [{0}] ", uriTemplate);
    HCatURI hcatURI;
    try {
        hcatURI = new HCatURI(uriTemplate.toString());
    }
    catch (URISyntaxException e) {
        LOG.info("uriTemplate [{0}]. Reason [{1}]: ", uriTemplate, e);
        throw new RuntimeException("HCat URI can't be parsed " + e);
    }
    return hcatURI;
}

这里并没有把所以的表达式都列全,但是大家应该可以了解了表达的计算模式,还有一点可能大家需要清除的,就是在实现方法中 有 ph1/ph2/ph3这样的标识,在ph1阶段可能仅仅是 把需要处理的数据放入上下文,在ph2或者ph3才会 真正计算出值,因为ph3、ph2的时候是action真正启动,实例的时候。

Oozie使用这些key 创建不同的 evaluator 从而到达 隔离的效果;

这里列举了不同的表达式,但是oozie在什么时候使用到他们的呢:
evaluator.evaluate

比如在 ActionStartXCommand 中:

String tmpActionConf = XmlUtils.removeComments(wfAction.getConf());
String actionConf = context.getELEvaluator().evaluate(tmpActionConf, String.class);

这样就可以将 actionConf 中所以的需要处理的表达式都处理了。

相关文章

  • Oozie-参数表达式体系

    ELService 服务是用于初始化el计算的服务,oozie的参数计算使用的是 org.apache.commo...

  • Oozie-监控体系-Instrumentation

    Oozie 第一个版的监控是自定义的,后面引进了 做监控当下主流的框架 Codahale Metrics 本文从o...

  • scala函数

    函数定义 默认参数 命名参数 可变参数 循环表达式 to | Range | until for循环

  • lamble表达式

    一、lambda表达式:匿名内部类的简写 1、两个参数: 2、一个参数 3、没有参数 二、lambda表达式:作用...

  • Linux的shell中 if 的-e,-d,-f的说明

    文件比较运算符 if [参数] For example 整数变量表达式if [参数] 字符串变量表达式 if [参...

  • 返回非空值函数LastnonBlank的第2参数使用方法

    LastnonBlank第2参数使用方法 第2参数是一个表达式,是对第一参数进行过滤的表达式。如果我们第二参数只写...

  • iOS Block

    Blocks 模式 Block语法 ^ 返回值类型 参数列表 表达式 ^ 参数列表 表达式 (省略返回值类型) ^...

  • [代码简化] 1 Lambda表达式

    1、lambda表达式 1.1 lambda 利用lambda表达式代替匿名参数类,将行为进行参数化传递到代码中,...

  • Java lambda表达式

    Lambda表达式是一种带有参数变量的表达式,参数变量可以不传;也可以将其视为一种匿名函数。Lambda表达式没有...

  • 快速了解箭头函数的简写

    核心概念:无论是一个表达式,还是一个参数,都可以省略括号 没有参数的情况 => 单个表达式,多个表达式 只有...

网友评论

      本文标题:Oozie-参数表达式体系

      本文链接:https://www.haomeiwen.com/subject/euncjttx.html