Recently, microgrid operation increased significantly with increasing distributed renewable energy resources in the power system. Microgrids can operate with and without utility. Fault currents are significantly different in island and utility connected operation modes. Therefore, microgrid protection is one of the important subjects in microgrid operation. In this paper, a hybrid particle swarm optimization (HPSO) approach has been developed for coordination of directional overcurrent relays (DOCRs) in a microgrid system. The coordination constraints include the utility connected and an autonomous condition of the microgrid operation. In the optimization procedure, the current setting (Iset) of relays is considered as discrete parameters and time multiplier settings (TMS) is assumed as continues parameter. The proposed algorithm has two parts, in the first part, PSO is used to calculate the Iset and in the second part, linear programming is applied to calculate the TMS of each relay. In the case study, loads connected to the network are divided into critical and noncritical ones. In normal operation of the system, distributed generators (DGs) operate in parallel with the utility. When a fault occurs on the utility side, noncritical loads are disconnected from the network and DGs are operated in microgrid as islanded mode. Regarding to simulation results, DOCRs have a suitable and reliable operation in both conditions of microgrid operations. In addition, overall operating time of the primary relays is properly minimized.